Sep 09 20:50:21.267562 ip-10-0-137-19 systemd[1]: Starting Kubernetes Kubelet... Sep 09 20:50:21.705581 ip-10-0-137-19 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 20:50:21.705581 ip-10-0-137-19 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 09 20:50:21.705581 ip-10-0-137-19 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 20:50:21.705581 ip-10-0-137-19 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 09 20:50:21.705581 ip-10-0-137-19 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 20:50:21.706551 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.706486 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 09 20:50:21.709196 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709181 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 20:50:21.709196 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709196 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 20:50:21.709256 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709200 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 20:50:21.709256 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709203 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 20:50:21.709256 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709206 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 20:50:21.709256 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709209 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 20:50:21.709256 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709212 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 20:50:21.709256 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709214 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 20:50:21.709256 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709218 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 20:50:21.709256 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709220 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 20:50:21.709256 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709223 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 20:50:21.709256 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709225 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 20:50:21.709256 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709228 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 20:50:21.709256 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709230 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 20:50:21.709256 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709233 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 20:50:21.709256 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709235 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 20:50:21.709256 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709238 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 20:50:21.709256 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709240 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 20:50:21.709256 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709243 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 20:50:21.709256 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709245 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 20:50:21.709256 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709254 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 20:50:21.709719 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709257 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 20:50:21.709719 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709260 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 20:50:21.709719 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709262 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 20:50:21.709719 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709265 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 20:50:21.709719 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709268 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 20:50:21.709719 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709271 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 20:50:21.709719 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709273 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 20:50:21.709719 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709276 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 20:50:21.709719 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709278 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 20:50:21.709719 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709282 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 20:50:21.709719 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709299 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 20:50:21.709719 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709302 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 20:50:21.709719 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709305 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 20:50:21.709719 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709308 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 20:50:21.709719 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709312 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 20:50:21.709719 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709315 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 20:50:21.709719 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709318 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 20:50:21.709719 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709320 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 20:50:21.709719 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709323 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 20:50:21.710159 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709326 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 20:50:21.710159 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709328 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 20:50:21.710159 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709330 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 20:50:21.710159 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709333 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 20:50:21.710159 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709336 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 20:50:21.710159 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709338 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 20:50:21.710159 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709341 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 20:50:21.710159 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709351 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 20:50:21.710159 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709354 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 20:50:21.710159 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709358 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 20:50:21.710159 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709362 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 20:50:21.710159 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709364 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 20:50:21.710159 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709367 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 20:50:21.710159 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709370 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 20:50:21.710159 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709373 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 20:50:21.710159 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709376 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 20:50:21.710159 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709379 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 20:50:21.710159 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709381 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 20:50:21.710159 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709383 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 20:50:21.710631 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709386 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 20:50:21.710631 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709388 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 20:50:21.710631 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709391 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 20:50:21.710631 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709393 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 20:50:21.710631 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709396 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 20:50:21.710631 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709398 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 20:50:21.710631 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709401 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 20:50:21.710631 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709403 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 20:50:21.710631 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709406 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 20:50:21.710631 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709408 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 20:50:21.710631 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709411 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 20:50:21.710631 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709415 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 20:50:21.710631 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709417 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 20:50:21.710631 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709419 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 20:50:21.710631 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709422 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 20:50:21.710631 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709424 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 20:50:21.710631 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709426 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 20:50:21.710631 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709429 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 20:50:21.710631 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709431 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 20:50:21.710631 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709433 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 20:50:21.711093 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709436 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 20:50:21.711093 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709438 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 20:50:21.711093 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709440 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 20:50:21.711093 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709443 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 20:50:21.711093 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709445 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 20:50:21.711093 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709447 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 20:50:21.711093 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709450 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 20:50:21.711093 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709453 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 20:50:21.711093 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709457 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 09 20:50:21.711093 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709459 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 20:50:21.711093 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709462 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 20:50:21.711093 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709464 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 20:50:21.711093 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709467 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 20:50:21.711093 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709469 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 20:50:21.711093 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709472 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 20:50:21.711093 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709474 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 20:50:21.711093 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709476 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 20:50:21.711093 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709479 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 20:50:21.711093 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709481 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 20:50:21.711093 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.709483 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 20:50:21.711614 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710140 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 20:50:21.711614 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710146 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 20:50:21.711614 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710150 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 20:50:21.711614 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710153 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 20:50:21.711614 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710156 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 20:50:21.711614 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710158 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 20:50:21.711614 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710161 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 20:50:21.711614 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710164 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 20:50:21.711614 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710167 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 20:50:21.711614 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710169 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 20:50:21.711614 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710172 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 20:50:21.711614 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710174 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 20:50:21.711614 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710177 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 20:50:21.711614 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710179 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 20:50:21.711614 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710182 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 20:50:21.711614 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710184 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 20:50:21.711614 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710187 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 20:50:21.711614 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710189 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 20:50:21.711614 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710191 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 20:50:21.711614 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710194 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 20:50:21.712091 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710198 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 09 20:50:21.712091 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710200 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 20:50:21.712091 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710203 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 20:50:21.712091 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710205 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 20:50:21.712091 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710208 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 20:50:21.712091 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710210 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 20:50:21.712091 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710213 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 20:50:21.712091 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710215 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 20:50:21.712091 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710217 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 20:50:21.712091 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710220 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 20:50:21.712091 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710222 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 20:50:21.712091 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710225 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 20:50:21.712091 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710227 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 20:50:21.712091 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710230 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 20:50:21.712091 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710232 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 20:50:21.712091 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710234 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 20:50:21.712091 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710237 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 20:50:21.712091 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710239 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 20:50:21.712091 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710241 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 20:50:21.712091 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710244 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 20:50:21.712583 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710246 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 20:50:21.712583 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710248 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 20:50:21.712583 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710251 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 20:50:21.712583 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710253 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 20:50:21.712583 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710257 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 20:50:21.712583 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710260 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 20:50:21.712583 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710263 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 20:50:21.712583 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710266 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 20:50:21.712583 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710268 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 20:50:21.712583 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710271 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 20:50:21.712583 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710273 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 20:50:21.712583 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710278 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 20:50:21.712583 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710281 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 20:50:21.712583 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710284 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 20:50:21.712583 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710303 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 20:50:21.712583 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710306 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 20:50:21.712583 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710308 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 20:50:21.712583 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710311 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 20:50:21.712583 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710313 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 20:50:21.712583 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710315 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 20:50:21.713048 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710319 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 20:50:21.713048 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710323 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 20:50:21.713048 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710326 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 20:50:21.713048 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710329 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 20:50:21.713048 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710331 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 20:50:21.713048 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710333 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 20:50:21.713048 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710336 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 20:50:21.713048 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710339 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 20:50:21.713048 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710342 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 20:50:21.713048 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710345 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 20:50:21.713048 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710348 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 20:50:21.713048 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710350 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 20:50:21.713048 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710354 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 20:50:21.713048 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710357 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 20:50:21.713048 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710360 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 20:50:21.713048 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710363 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 20:50:21.713048 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710366 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 20:50:21.713048 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710368 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 20:50:21.713048 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710371 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 20:50:21.713514 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710373 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 20:50:21.713514 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710376 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 20:50:21.713514 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710378 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 20:50:21.713514 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710380 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 20:50:21.713514 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710383 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 20:50:21.713514 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710386 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 20:50:21.713514 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710388 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 20:50:21.713514 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710391 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 20:50:21.713514 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710394 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 20:50:21.713514 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710397 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 20:50:21.713514 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710399 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 20:50:21.713514 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710401 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 20:50:21.713514 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710404 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 20:50:21.713514 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710406 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 20:50:21.713514 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710408 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 20:50:21.713514 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710411 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 20:50:21.713514 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710413 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 20:50:21.713514 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710416 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 20:50:21.713514 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710418 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 20:50:21.713514 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710420 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 20:50:21.713983 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710742 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 20:50:21.713983 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710747 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 20:50:21.713983 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710750 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 20:50:21.713983 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710752 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 20:50:21.713983 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710755 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 20:50:21.713983 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710757 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 20:50:21.713983 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710760 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 20:50:21.713983 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710762 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 20:50:21.713983 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710765 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 20:50:21.713983 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710767 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 20:50:21.713983 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710770 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 20:50:21.713983 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710772 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 20:50:21.713983 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710775 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 20:50:21.713983 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710777 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 20:50:21.713983 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710779 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 20:50:21.713983 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710782 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 20:50:21.713983 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710785 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 20:50:21.713983 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710788 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 20:50:21.713983 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710790 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 20:50:21.713983 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710794 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 20:50:21.714482 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710796 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 20:50:21.714482 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710798 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 20:50:21.714482 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710801 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 20:50:21.714482 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710803 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 20:50:21.714482 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710806 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 20:50:21.714482 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710808 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 20:50:21.714482 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710811 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 20:50:21.714482 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710813 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 20:50:21.714482 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710817 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 20:50:21.714482 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710821 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 20:50:21.714482 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710824 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 20:50:21.714482 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710827 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 20:50:21.714482 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710830 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 20:50:21.714482 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710832 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 20:50:21.714482 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710835 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 20:50:21.714482 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710838 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 20:50:21.714482 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710840 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 20:50:21.714482 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710842 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 20:50:21.714482 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710845 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 20:50:21.714920 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710847 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 20:50:21.714920 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710850 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 20:50:21.714920 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710852 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 20:50:21.714920 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710854 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 20:50:21.714920 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710857 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 09 20:50:21.714920 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710859 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 20:50:21.714920 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710861 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 20:50:21.714920 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710864 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 20:50:21.714920 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710866 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 20:50:21.714920 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710869 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 20:50:21.714920 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710872 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 20:50:21.714920 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710874 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 20:50:21.714920 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710876 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 20:50:21.714920 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710880 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 20:50:21.714920 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710882 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 20:50:21.714920 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710885 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 20:50:21.714920 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710887 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 20:50:21.714920 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710890 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 20:50:21.714920 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710892 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 20:50:21.714920 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710895 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 20:50:21.714920 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710897 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 20:50:21.715561 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710899 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 20:50:21.715561 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710902 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 20:50:21.715561 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710904 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 20:50:21.715561 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710906 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 20:50:21.715561 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710909 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 20:50:21.715561 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710911 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 20:50:21.715561 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710913 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 20:50:21.715561 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710916 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 20:50:21.715561 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710918 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 20:50:21.715561 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710921 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 20:50:21.715561 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710923 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 20:50:21.715561 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710926 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 20:50:21.715561 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710928 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 20:50:21.715561 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710931 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 20:50:21.715561 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710934 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 20:50:21.715561 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710936 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 20:50:21.715561 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710939 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 20:50:21.715561 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710941 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 20:50:21.715561 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710943 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 20:50:21.715561 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710948 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 20:50:21.716040 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710951 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 20:50:21.716040 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710953 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 20:50:21.716040 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710956 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 20:50:21.716040 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710958 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 20:50:21.716040 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710961 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 20:50:21.716040 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710963 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 20:50:21.716040 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710966 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 20:50:21.716040 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710968 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 20:50:21.716040 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710970 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 20:50:21.716040 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710973 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 20:50:21.716040 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710975 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 20:50:21.716040 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710978 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 20:50:21.716040 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710980 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 20:50:21.716040 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710982 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 20:50:21.716040 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710985 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 20:50:21.716040 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710987 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 20:50:21.716040 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710989 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 20:50:21.716040 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710992 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 20:50:21.716040 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.710994 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 20:50:21.716040 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711074 2580 flags.go:64] FLAG: --address="0.0.0.0" Sep 09 20:50:21.716559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711118 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 09 20:50:21.716559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711125 2580 flags.go:64] FLAG: --anonymous-auth="true" Sep 09 20:50:21.716559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711129 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 09 20:50:21.716559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711133 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 09 20:50:21.716559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711136 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 09 20:50:21.716559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711140 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 09 20:50:21.716559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711144 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 09 20:50:21.716559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711147 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 09 20:50:21.716559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711150 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 09 20:50:21.716559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711153 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 09 20:50:21.716559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711156 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 09 20:50:21.716559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711161 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 09 20:50:21.716559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711165 2580 flags.go:64] FLAG: --cgroup-root="" Sep 09 20:50:21.716559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711168 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 09 20:50:21.716559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711171 2580 flags.go:64] FLAG: --client-ca-file="" Sep 09 20:50:21.716559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711174 2580 flags.go:64] FLAG: --cloud-provider="external" Sep 09 20:50:21.716559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711177 2580 flags.go:64] FLAG: --cluster-dns="[]" Sep 09 20:50:21.716559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711181 2580 flags.go:64] FLAG: --cluster-domain="" Sep 09 20:50:21.716559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711184 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 09 20:50:21.716559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711187 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 09 20:50:21.716559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711190 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 09 20:50:21.716559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711193 2580 flags.go:64] FLAG: --container-log-max-files="5" Sep 09 20:50:21.716559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711197 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 09 20:50:21.716559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711199 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 09 20:50:21.717114 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711202 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 09 20:50:21.717114 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711205 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 09 20:50:21.717114 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711208 2580 flags.go:64] FLAG: --contention-profiling="false" Sep 09 20:50:21.717114 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711211 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 09 20:50:21.717114 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711213 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 09 20:50:21.717114 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711216 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 09 20:50:21.717114 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711222 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 09 20:50:21.717114 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711225 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 09 20:50:21.717114 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711228 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 09 20:50:21.717114 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711231 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 09 20:50:21.717114 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711233 2580 flags.go:64] FLAG: --enable-load-reader="false" Sep 09 20:50:21.717114 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711236 2580 flags.go:64] FLAG: --enable-server="true" Sep 09 20:50:21.717114 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711239 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 09 20:50:21.717114 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711242 2580 flags.go:64] FLAG: --event-burst="100" Sep 09 20:50:21.717114 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711245 2580 flags.go:64] FLAG: --event-qps="50" Sep 09 20:50:21.717114 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711248 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 09 20:50:21.717114 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711253 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 09 20:50:21.717114 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711256 2580 flags.go:64] FLAG: --eviction-hard="" Sep 09 20:50:21.717114 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711259 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 09 20:50:21.717114 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711263 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 09 20:50:21.717114 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711266 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 09 20:50:21.717114 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711269 2580 flags.go:64] FLAG: --eviction-soft="" Sep 09 20:50:21.717114 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711271 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 09 20:50:21.717114 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711274 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 09 20:50:21.717114 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711276 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 09 20:50:21.717748 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711279 2580 flags.go:64] FLAG: --experimental-mounter-path="" Sep 09 20:50:21.717748 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711282 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 09 20:50:21.717748 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711299 2580 flags.go:64] FLAG: --fail-swap-on="true" Sep 09 20:50:21.717748 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711303 2580 flags.go:64] FLAG: --feature-gates="" Sep 09 20:50:21.717748 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711306 2580 flags.go:64] FLAG: --file-check-frequency="20s" Sep 09 20:50:21.717748 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711309 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 09 20:50:21.717748 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711312 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 09 20:50:21.717748 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711315 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 09 20:50:21.717748 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711318 2580 flags.go:64] FLAG: --healthz-port="10248" Sep 09 20:50:21.717748 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711321 2580 flags.go:64] FLAG: --help="false" Sep 09 20:50:21.717748 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711324 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-137-19.ec2.internal" Sep 09 20:50:21.717748 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711326 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 09 20:50:21.717748 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711329 2580 flags.go:64] FLAG: --http-check-frequency="20s" Sep 09 20:50:21.717748 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711332 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 09 20:50:21.717748 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711335 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 09 20:50:21.717748 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711338 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 09 20:50:21.717748 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711340 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 09 20:50:21.717748 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711343 2580 flags.go:64] FLAG: --image-service-endpoint="" Sep 09 20:50:21.717748 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711346 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 09 20:50:21.717748 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711348 2580 flags.go:64] FLAG: --kube-api-burst="100" Sep 09 20:50:21.717748 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711351 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 09 20:50:21.717748 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711354 2580 flags.go:64] FLAG: --kube-api-qps="50" Sep 09 20:50:21.717748 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711356 2580 flags.go:64] FLAG: --kube-reserved="" Sep 09 20:50:21.717748 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711360 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 09 20:50:21.718459 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711363 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 09 20:50:21.718459 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711366 2580 flags.go:64] FLAG: --kubelet-cgroups="" Sep 09 20:50:21.718459 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711370 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 09 20:50:21.718459 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711373 2580 flags.go:64] FLAG: --lock-file="" Sep 09 20:50:21.718459 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711376 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 09 20:50:21.718459 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711378 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 09 20:50:21.718459 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711381 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 09 20:50:21.718459 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711386 2580 flags.go:64] FLAG: --log-json-split-stream="false" Sep 09 20:50:21.718459 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711389 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 09 20:50:21.718459 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711391 2580 flags.go:64] FLAG: --log-text-split-stream="false" Sep 09 20:50:21.718459 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711394 2580 flags.go:64] FLAG: --logging-format="text" Sep 09 20:50:21.718459 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711397 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 09 20:50:21.718459 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711400 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 09 20:50:21.718459 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711402 2580 flags.go:64] FLAG: --manifest-url="" Sep 09 20:50:21.718459 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711405 2580 flags.go:64] FLAG: --manifest-url-header="" Sep 09 20:50:21.718459 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711409 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 09 20:50:21.718459 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711412 2580 flags.go:64] FLAG: --max-open-files="1000000" Sep 09 20:50:21.718459 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711416 2580 flags.go:64] FLAG: --max-pods="110" Sep 09 20:50:21.718459 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711418 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 09 20:50:21.718459 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711421 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 09 20:50:21.718459 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711423 2580 flags.go:64] FLAG: --memory-manager-policy="None" Sep 09 20:50:21.718459 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711426 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 09 20:50:21.718459 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711429 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 09 20:50:21.718459 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711431 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 09 20:50:21.718459 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711434 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 09 20:50:21.719199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711440 2580 flags.go:64] FLAG: --node-status-max-images="50" Sep 09 20:50:21.719199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711443 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 09 20:50:21.719199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711446 2580 flags.go:64] FLAG: --oom-score-adj="-999" Sep 09 20:50:21.719199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711449 2580 flags.go:64] FLAG: --pod-cidr="" Sep 09 20:50:21.719199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711451 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 09 20:50:21.719199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711457 2580 flags.go:64] FLAG: --pod-manifest-path="" Sep 09 20:50:21.719199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711459 2580 flags.go:64] FLAG: --pod-max-pids="-1" Sep 09 20:50:21.719199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711463 2580 flags.go:64] FLAG: --pods-per-core="0" Sep 09 20:50:21.719199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711466 2580 flags.go:64] FLAG: --port="10250" Sep 09 20:50:21.719199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711470 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 09 20:50:21.719199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711473 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05a05916a821efed1" Sep 09 20:50:21.719199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711476 2580 flags.go:64] FLAG: --qos-reserved="" Sep 09 20:50:21.719199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711479 2580 flags.go:64] FLAG: --read-only-port="10255" Sep 09 20:50:21.719199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711481 2580 flags.go:64] FLAG: --register-node="true" Sep 09 20:50:21.719199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711484 2580 flags.go:64] FLAG: --register-with-taints="" Sep 09 20:50:21.719199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711487 2580 flags.go:64] FLAG: --registry-burst="10" Sep 09 20:50:21.719199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711490 2580 flags.go:64] FLAG: --registry-qps="5" Sep 09 20:50:21.719199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711493 2580 flags.go:64] FLAG: --reserved-cpus="" Sep 09 20:50:21.719199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711495 2580 flags.go:64] FLAG: --reserved-memory="" Sep 09 20:50:21.719199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711499 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 09 20:50:21.719199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711502 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 09 20:50:21.719199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711505 2580 flags.go:64] FLAG: --rotate-certificates="false" Sep 09 20:50:21.719199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711508 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 09 20:50:21.719199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711510 2580 flags.go:64] FLAG: --runonce="false" Sep 09 20:50:21.719199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711513 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 09 20:50:21.720128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711516 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 09 20:50:21.720128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711518 2580 flags.go:64] FLAG: --seccomp-default="false" Sep 09 20:50:21.720128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711521 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 09 20:50:21.720128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711524 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 09 20:50:21.720128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711527 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 09 20:50:21.720128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711529 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 09 20:50:21.720128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711532 2580 flags.go:64] FLAG: --storage-driver-password="root" Sep 09 20:50:21.720128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711535 2580 flags.go:64] FLAG: --storage-driver-secure="false" Sep 09 20:50:21.720128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711537 2580 flags.go:64] FLAG: --storage-driver-table="stats" Sep 09 20:50:21.720128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711540 2580 flags.go:64] FLAG: --storage-driver-user="root" Sep 09 20:50:21.720128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711542 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 09 20:50:21.720128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711545 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 09 20:50:21.720128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711548 2580 flags.go:64] FLAG: --system-cgroups="" Sep 09 20:50:21.720128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711550 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 09 20:50:21.720128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711557 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 09 20:50:21.720128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711559 2580 flags.go:64] FLAG: --tls-cert-file="" Sep 09 20:50:21.720128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711563 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 09 20:50:21.720128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711568 2580 flags.go:64] FLAG: --tls-min-version="" Sep 09 20:50:21.720128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711570 2580 flags.go:64] FLAG: --tls-private-key-file="" Sep 09 20:50:21.720128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711573 2580 flags.go:64] FLAG: --topology-manager-policy="none" Sep 09 20:50:21.720128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711575 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 09 20:50:21.720128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711578 2580 flags.go:64] FLAG: --topology-manager-scope="container" Sep 09 20:50:21.720128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711582 2580 flags.go:64] FLAG: --v="2" Sep 09 20:50:21.720128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711585 2580 flags.go:64] FLAG: --version="false" Sep 09 20:50:21.720128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711589 2580 flags.go:64] FLAG: --vmodule="" Sep 09 20:50:21.721222 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711593 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 09 20:50:21.721222 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.711596 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 09 20:50:21.721222 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711699 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 20:50:21.721222 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711703 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 20:50:21.721222 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711705 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 20:50:21.721222 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711708 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 20:50:21.721222 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711710 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 20:50:21.721222 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711713 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 20:50:21.721222 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711716 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 20:50:21.721222 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711718 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 20:50:21.721222 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711721 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 20:50:21.721222 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711724 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 20:50:21.721222 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711728 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 20:50:21.721222 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711731 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 20:50:21.721222 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711734 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 20:50:21.721222 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711736 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 20:50:21.721222 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711739 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 20:50:21.721222 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711741 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 20:50:21.721222 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711744 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 20:50:21.721222 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711747 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 20:50:21.722142 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711750 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 20:50:21.722142 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711753 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 20:50:21.722142 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711756 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 09 20:50:21.722142 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711759 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 20:50:21.722142 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711762 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 20:50:21.722142 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711764 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 20:50:21.722142 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711767 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 20:50:21.722142 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711770 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 20:50:21.722142 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711773 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 20:50:21.722142 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711775 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 20:50:21.722142 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711777 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 20:50:21.722142 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711780 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 20:50:21.722142 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711782 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 20:50:21.722142 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711784 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 20:50:21.722142 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711787 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 20:50:21.722142 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711790 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 20:50:21.722142 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711792 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 20:50:21.722142 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711794 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 20:50:21.722142 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711797 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 20:50:21.722142 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711800 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 20:50:21.722697 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711802 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 20:50:21.722697 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711804 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 20:50:21.722697 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711807 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 20:50:21.722697 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711809 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 20:50:21.722697 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711812 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 20:50:21.722697 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711814 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 20:50:21.722697 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711816 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 20:50:21.722697 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711818 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 20:50:21.722697 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711821 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 20:50:21.722697 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711823 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 20:50:21.722697 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711825 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 20:50:21.722697 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711828 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 20:50:21.722697 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711830 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 20:50:21.722697 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711833 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 20:50:21.722697 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711835 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 20:50:21.722697 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711838 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 20:50:21.722697 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711841 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 20:50:21.722697 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711843 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 20:50:21.722697 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711846 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 20:50:21.723479 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711848 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 20:50:21.723479 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711851 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 20:50:21.723479 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711853 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 20:50:21.723479 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711855 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 20:50:21.723479 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711858 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 20:50:21.723479 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711861 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 20:50:21.723479 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711863 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 20:50:21.723479 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711865 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 20:50:21.723479 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711867 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 20:50:21.723479 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711871 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 20:50:21.723479 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711875 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 20:50:21.723479 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711878 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 20:50:21.723479 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711880 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 20:50:21.723479 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711883 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 20:50:21.723479 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711886 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 20:50:21.723479 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711888 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 20:50:21.723479 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711891 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 20:50:21.723479 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711893 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 20:50:21.723479 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711895 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 20:50:21.723479 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711898 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 20:50:21.724319 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711900 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 20:50:21.724319 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711902 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 20:50:21.724319 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711905 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 20:50:21.724319 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711907 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 20:50:21.724319 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711909 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 20:50:21.724319 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711912 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 20:50:21.724319 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711914 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 20:50:21.724319 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711917 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 20:50:21.724319 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711919 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 20:50:21.724319 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711921 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 20:50:21.724319 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711924 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 20:50:21.724319 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711926 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 20:50:21.724319 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711929 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 20:50:21.724319 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711931 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 20:50:21.724319 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711933 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 20:50:21.724319 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711936 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 20:50:21.724319 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711938 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 20:50:21.724319 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711940 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 20:50:21.724319 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711943 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 20:50:21.724319 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711945 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 20:50:21.724878 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711948 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 20:50:21.724878 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.711950 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 20:50:21.724878 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.712481 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 20:50:21.724878 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.720678 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 09 20:50:21.724878 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.720705 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 09 20:50:21.724878 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.720722 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 09 20:50:21.724878 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.720730 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 09 20:50:21.724878 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720811 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 20:50:21.724878 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720819 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 20:50:21.724878 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720825 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 20:50:21.724878 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720830 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 20:50:21.724878 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720834 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 20:50:21.724878 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720839 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 20:50:21.724878 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720843 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 20:50:21.724878 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720847 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 20:50:21.725235 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720851 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 20:50:21.725235 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720855 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 20:50:21.725235 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720859 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 20:50:21.725235 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720863 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 20:50:21.725235 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720867 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 20:50:21.725235 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720871 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 09 20:50:21.725235 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720876 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 20:50:21.725235 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720880 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 20:50:21.725235 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720884 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 20:50:21.725235 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720888 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 20:50:21.725235 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720892 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 20:50:21.725235 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720898 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 20:50:21.725235 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720905 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 20:50:21.725235 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720910 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 20:50:21.725235 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720914 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 20:50:21.725235 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720918 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 20:50:21.725235 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720922 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 20:50:21.725235 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720926 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 20:50:21.725235 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720930 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 20:50:21.725235 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720934 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 20:50:21.725827 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720938 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 20:50:21.725827 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720943 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 20:50:21.725827 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720947 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 20:50:21.725827 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720951 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 20:50:21.725827 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720955 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 20:50:21.725827 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720958 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 20:50:21.725827 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720963 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 20:50:21.725827 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720967 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 20:50:21.725827 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720970 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 20:50:21.725827 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720974 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 20:50:21.725827 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720978 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 20:50:21.725827 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720982 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 20:50:21.725827 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720986 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 20:50:21.725827 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720990 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 20:50:21.725827 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720994 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 20:50:21.725827 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.720998 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 20:50:21.725827 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721002 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 20:50:21.725827 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721006 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 20:50:21.725827 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721011 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 20:50:21.725827 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721014 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 20:50:21.726323 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721018 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 20:50:21.726323 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721023 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 20:50:21.726323 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721027 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 20:50:21.726323 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721031 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 20:50:21.726323 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721035 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 20:50:21.726323 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721039 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 20:50:21.726323 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721043 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 20:50:21.726323 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721047 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 20:50:21.726323 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721051 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 20:50:21.726323 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721055 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 20:50:21.726323 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721058 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 20:50:21.726323 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721063 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 20:50:21.726323 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721067 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 20:50:21.726323 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721072 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 20:50:21.726323 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721078 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 20:50:21.726323 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721084 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 20:50:21.726323 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721088 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 20:50:21.726323 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721093 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 20:50:21.726323 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721097 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 20:50:21.726817 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721101 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 20:50:21.726817 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721107 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 20:50:21.726817 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721112 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 20:50:21.726817 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721117 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 20:50:21.726817 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721121 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 20:50:21.726817 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721125 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 20:50:21.726817 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721130 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 20:50:21.726817 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721134 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 20:50:21.726817 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721138 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 20:50:21.726817 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721142 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 20:50:21.726817 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721146 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 20:50:21.726817 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721150 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 20:50:21.726817 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721155 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 20:50:21.726817 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721159 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 20:50:21.726817 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721163 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 20:50:21.726817 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721167 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 20:50:21.726817 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721171 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 20:50:21.726817 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721175 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 20:50:21.726817 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721178 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 20:50:21.726817 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721182 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 20:50:21.727305 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721186 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 20:50:21.727305 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721190 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 20:50:21.727305 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721194 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 20:50:21.727305 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721197 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 20:50:21.727305 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721201 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 20:50:21.727305 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721205 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 20:50:21.727305 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721210 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 20:50:21.727305 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721214 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 20:50:21.727305 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721218 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 20:50:21.727305 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721221 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 20:50:21.727305 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721225 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 20:50:21.727305 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721228 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 20:50:21.727305 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.721235 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 20:50:21.727305 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721426 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 20:50:21.727305 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721436 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 20:50:21.727699 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721441 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 20:50:21.727699 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721445 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 20:50:21.727699 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721450 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 20:50:21.727699 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721454 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 20:50:21.727699 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721459 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 20:50:21.727699 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721463 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 20:50:21.727699 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721468 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 20:50:21.727699 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721472 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 20:50:21.727699 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721476 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 20:50:21.727699 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721480 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 20:50:21.727699 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721485 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 20:50:21.727699 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721489 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 20:50:21.727699 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721493 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 20:50:21.727699 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721498 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 20:50:21.727699 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721502 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 20:50:21.727699 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721506 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 20:50:21.727699 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721510 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 20:50:21.727699 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721514 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 20:50:21.727699 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721517 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 20:50:21.727699 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721521 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 20:50:21.728225 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721525 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 20:50:21.728225 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721529 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 20:50:21.728225 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721533 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 20:50:21.728225 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721537 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 20:50:21.728225 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721541 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 20:50:21.728225 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721545 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 20:50:21.728225 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721549 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 20:50:21.728225 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721555 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 20:50:21.728225 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721559 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 20:50:21.728225 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721563 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 20:50:21.728225 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721567 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 20:50:21.728225 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721571 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 20:50:21.728225 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721576 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 20:50:21.728225 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721579 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 20:50:21.728225 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721583 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 20:50:21.728225 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721587 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 20:50:21.728225 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721591 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 20:50:21.728225 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721595 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 20:50:21.728225 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721599 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 20:50:21.728225 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721602 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 20:50:21.729039 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721606 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 20:50:21.729039 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721610 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 20:50:21.729039 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721614 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 20:50:21.729039 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721619 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 20:50:21.729039 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721623 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 20:50:21.729039 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721627 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 20:50:21.729039 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721631 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 20:50:21.729039 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721635 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 20:50:21.729039 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721638 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 20:50:21.729039 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721642 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 20:50:21.729039 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721646 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 20:50:21.729039 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721650 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 09 20:50:21.729039 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721655 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 20:50:21.729039 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721658 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 20:50:21.729039 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721662 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 20:50:21.729039 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721666 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 20:50:21.729039 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721670 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 20:50:21.729039 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721674 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 20:50:21.729039 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721678 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 20:50:21.729039 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721682 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 20:50:21.729552 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721686 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 20:50:21.729552 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721691 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 20:50:21.729552 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721695 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 20:50:21.729552 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721699 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 20:50:21.729552 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721703 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 20:50:21.729552 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721706 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 20:50:21.729552 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721710 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 20:50:21.729552 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721714 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 20:50:21.729552 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721720 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 20:50:21.729552 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721725 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 20:50:21.729552 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721730 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 20:50:21.729552 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721735 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 20:50:21.729552 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721739 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 20:50:21.729552 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721744 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 20:50:21.729552 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721748 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 20:50:21.729552 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721752 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 20:50:21.729552 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721758 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 20:50:21.729552 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721764 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 20:50:21.729552 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721769 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 20:50:21.730008 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721773 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 20:50:21.730008 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721777 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 20:50:21.730008 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721781 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 20:50:21.730008 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721787 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 20:50:21.730008 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721792 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 20:50:21.730008 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721796 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 20:50:21.730008 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721801 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 20:50:21.730008 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721805 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 20:50:21.730008 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721808 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 20:50:21.730008 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721813 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 20:50:21.730008 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721817 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 20:50:21.730008 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721821 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 20:50:21.730008 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721825 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 20:50:21.730008 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721829 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 20:50:21.730008 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721833 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 20:50:21.730008 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721838 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 20:50:21.730008 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721841 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 20:50:21.730008 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:21.721845 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 20:50:21.730451 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.721853 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 20:50:21.730451 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.722000 2580 server.go:962] "Client rotation is on, will bootstrap in background" Sep 09 20:50:21.730451 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.724268 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 09 20:50:21.730451 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.725534 2580 server.go:1019] "Starting client certificate rotation" Sep 09 20:50:21.730451 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.725646 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 20:50:21.730451 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.726404 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 20:50:21.730451 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.728765 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 20:50:21.731722 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.731702 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 20:50:21.740351 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.740336 2580 log.go:25] "Validated CRI v1 runtime API" Sep 09 20:50:21.746781 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.746766 2580 log.go:25] "Validated CRI v1 image API" Sep 09 20:50:21.747702 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.747676 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 09 20:50:21.752041 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.752018 2580 fs.go:135] Filesystem UUIDs: map[3b0df51b-1416-4614-9890-94f3c77fcf1b:/dev/nvme0n1p3 72c453de-e6aa-485b-ab6b-2313079ea4c3:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Sep 09 20:50:21.752102 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.752041 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 09 20:50:21.758614 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.758501 2580 manager.go:216] Machine: {Timestamp:2026-09-09 20:50:21.755985496 +0000 UTC m=+0.362228196 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200140 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b2e4467631572495fe14bca4bbb57 SystemUUID:ec2b2e44-6763-1572-495f-e14bca4bbb57 BootID:f3cb07d6-d35a-49a1-a852-ea3bf5d7ed6e Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a2:a4:29:90:8b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a2:a4:29:90:8b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:ac:0b:c7:5c:3b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 09 20:50:21.758614 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.758612 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 09 20:50:21.758723 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.758707 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 09 20:50:21.759374 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.759361 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 20:50:21.762449 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.762430 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 09 20:50:21.762975 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.762454 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-19.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 09 20:50:21.763687 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.763667 2580 topology_manager.go:138] "Creating topology manager with none policy" Sep 09 20:50:21.763765 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.763693 2580 container_manager_linux.go:309] "Creating device plugin manager" Sep 09 20:50:21.763765 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.763704 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 20:50:21.763765 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.763754 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 20:50:21.764657 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.764644 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 09 20:50:21.768374 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.768363 2580 state_mem.go:36] "Initialized new in-memory state store" Sep 09 20:50:21.768475 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.768466 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 09 20:50:21.768507 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.768494 2580 kubelet.go:486] "Attempting to sync node with API server" Sep 09 20:50:21.768507 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.768503 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 09 20:50:21.768560 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.768517 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 09 20:50:21.768560 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.768526 2580 kubelet.go:398] "Adding apiserver pod source" Sep 09 20:50:21.768560 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.768533 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 09 20:50:21.768923 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.768909 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 20:50:21.768982 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.768935 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 20:50:21.771148 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.771134 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 09 20:50:21.773204 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.773192 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 09 20:50:21.773237 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.773221 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 09 20:50:21.773937 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.773918 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 09 20:50:21.773937 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.773935 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 09 20:50:21.773937 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.773941 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 09 20:50:21.774073 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.773946 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 09 20:50:21.774073 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.773953 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 09 20:50:21.774073 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.773958 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 09 20:50:21.774073 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.774045 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 09 20:50:21.774073 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.774050 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 09 20:50:21.774205 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.774076 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 09 20:50:21.774205 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.774085 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 09 20:50:21.774205 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.774097 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 09 20:50:21.774205 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.774107 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 09 20:50:21.774986 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.774976 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 09 20:50:21.774986 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.774985 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 09 20:50:21.778592 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.778575 2580 server.go:1268] "Started kubelet" Sep 09 20:50:21.778697 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.778667 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 09 20:50:21.778761 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:21.778727 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 09 20:50:21.778813 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.778800 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 09 20:50:21.778930 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:21.778907 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 09 20:50:21.779012 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.778655 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 09 20:50:21.779118 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.779095 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 09 20:50:21.779392 ip-10-0-137-19 systemd[1]: Started Kubernetes Kubelet. Sep 09 20:50:21.779714 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.779687 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4bzm6" Sep 09 20:50:21.780384 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.780346 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 09 20:50:21.780968 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.780955 2580 server.go:310] "Adding debug handlers to kubelet server" Sep 09 20:50:21.783875 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:21.783856 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 09 20:50:21.784327 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.784312 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 09 20:50:21.784383 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.784325 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 09 20:50:21.784432 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:21.784423 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-19.ec2.internal\" not found" Sep 09 20:50:21.784924 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.784903 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 09 20:50:21.785023 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.784906 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 09 20:50:21.785023 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.784941 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 09 20:50:21.785147 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.785062 2580 reconstruct.go:97] "Volume reconstruction finished" Sep 09 20:50:21.785147 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.785070 2580 reconciler.go:29] "Reconciler: start to sync state" Sep 09 20:50:21.785442 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.785422 2580 factory.go:55] Registering systemd factory Sep 09 20:50:21.785442 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.785442 2580 factory.go:223] Registration of the systemd container factory successfully Sep 09 20:50:21.785902 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.785886 2580 factory.go:153] Registering CRI-O factory Sep 09 20:50:21.785977 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.785904 2580 factory.go:223] Registration of the crio container factory successfully Sep 09 20:50:21.785977 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.785948 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 09 20:50:21.785977 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.785961 2580 factory.go:103] Registering Raw factory Sep 09 20:50:21.785977 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.785971 2580 manager.go:1196] Started watching for new ooms in manager Sep 09 20:50:21.786141 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:21.785137 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-19.ec2.internal.18d3c237c4e4fc34 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-19.ec2.internal,UID:ip-10-0-137-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-19.ec2.internal,},FirstTimestamp:2026-09-09 20:50:21.778549812 +0000 UTC m=+0.384792510,LastTimestamp:2026-09-09 20:50:21.778549812 +0000 UTC m=+0.384792510,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-19.ec2.internal,}" Sep 09 20:50:21.786273 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.786261 2580 manager.go:349] Starting recovery of all containers Sep 09 20:50:21.789180 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.788823 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4bzm6" Sep 09 20:50:21.789180 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:21.788908 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 09 20:50:21.789180 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:21.789022 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 09 20:50:21.798367 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.798353 2580 manager.go:354] Recovery completed Sep 09 20:50:21.799821 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:21.799798 2580 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Sep 09 20:50:21.802554 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.802543 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 20:50:21.804253 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.804240 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientMemory" Sep 09 20:50:21.804342 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.804263 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 20:50:21.804342 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.804273 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientPID" Sep 09 20:50:21.805441 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.805389 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 09 20:50:21.805441 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.805404 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 09 20:50:21.805441 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.805422 2580 state_mem.go:36] "Initialized new in-memory state store" Sep 09 20:50:21.805614 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.805420 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 20:50:21.806200 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:21.806127 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-19.ec2.internal.18d3c237c66d2d38 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-19.ec2.internal,UID:ip-10-0-137-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-19.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-19.ec2.internal,},FirstTimestamp:2026-09-09 20:50:21.804252472 +0000 UTC m=+0.410495171,LastTimestamp:2026-09-09 20:50:21.804252472 +0000 UTC m=+0.410495171,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-19.ec2.internal,}" Sep 09 20:50:21.806282 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.806251 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientMemory" Sep 09 20:50:21.806282 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.806272 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 20:50:21.806372 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.806284 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientPID" Sep 09 20:50:21.807539 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.807524 2580 policy_none.go:49] "None policy: Start" Sep 09 20:50:21.807613 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.807542 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 09 20:50:21.807613 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.807557 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 09 20:50:21.809578 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.809565 2580 policy_none.go:47] "Start" Sep 09 20:50:21.850895 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.850880 2580 manager.go:337] "Starting Device Plugin manager" Sep 09 20:50:21.869025 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:21.850914 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 09 20:50:21.869025 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.850929 2580 server.go:87] "Starting device plugin registration server" Sep 09 20:50:21.869025 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.851148 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 09 20:50:21.869025 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.851158 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 09 20:50:21.869025 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.851253 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 09 20:50:21.869025 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.851418 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 09 20:50:21.869025 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.851427 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 09 20:50:21.869025 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:21.851773 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 09 20:50:21.869025 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:21.851815 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-19.ec2.internal\" not found" Sep 09 20:50:21.890209 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.890189 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 09 20:50:21.891570 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.891556 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 09 20:50:21.891650 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.891603 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 09 20:50:21.891650 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.891627 2580 kubelet.go:2448] "Starting kubelet main sync loop" Sep 09 20:50:21.891751 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:21.891697 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 09 20:50:21.894512 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.894497 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 20:50:21.952119 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.952102 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 20:50:21.952796 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.952781 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientMemory" Sep 09 20:50:21.952851 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.952808 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 20:50:21.952851 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.952823 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientPID" Sep 09 20:50:21.952851 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.952846 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-19.ec2.internal" Sep 09 20:50:21.961899 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.961842 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-19.ec2.internal" Sep 09 20:50:21.961899 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:21.961864 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-19.ec2.internal\": node \"ip-10-0-137-19.ec2.internal\" not found" Sep 09 20:50:21.979437 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:21.979415 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-19.ec2.internal\" not found" Sep 09 20:50:21.992771 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.992752 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-19.ec2.internal"] Sep 09 20:50:21.992838 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.992803 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 20:50:21.993490 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.993474 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientMemory" Sep 09 20:50:21.993564 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.993504 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 20:50:21.993564 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.993517 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientPID" Sep 09 20:50:21.996154 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.996140 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 20:50:21.996277 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.996262 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal" Sep 09 20:50:21.996342 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.996305 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 20:50:21.996808 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.996794 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientMemory" Sep 09 20:50:21.996866 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.996816 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 20:50:21.996866 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.996827 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientPID" Sep 09 20:50:21.996866 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.996862 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientMemory" Sep 09 20:50:21.996969 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.996878 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 20:50:21.996969 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.996891 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientPID" Sep 09 20:50:21.999422 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.999407 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-19.ec2.internal" Sep 09 20:50:21.999491 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:21.999428 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 20:50:22.000116 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.000100 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientMemory" Sep 09 20:50:22.000199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.000125 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 20:50:22.000199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.000134 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientPID" Sep 09 20:50:22.028839 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:22.028819 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-19.ec2.internal\" not found" node="ip-10-0-137-19.ec2.internal" Sep 09 20:50:22.032889 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:22.032873 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-19.ec2.internal\" not found" node="ip-10-0-137-19.ec2.internal" Sep 09 20:50:22.080194 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:22.080173 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-19.ec2.internal\" not found" Sep 09 20:50:22.087561 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.087543 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99981a6272193d33eff6882647ea56fb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal\" (UID: \"99981a6272193d33eff6882647ea56fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal" Sep 09 20:50:22.087631 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.087563 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70b70169d86c3bcf2d0a1e114ed46d9a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-19.ec2.internal\" (UID: \"70b70169d86c3bcf2d0a1e114ed46d9a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-19.ec2.internal" Sep 09 20:50:22.087631 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.087579 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99981a6272193d33eff6882647ea56fb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal\" (UID: \"99981a6272193d33eff6882647ea56fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal" Sep 09 20:50:22.180514 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:22.180496 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-19.ec2.internal\" not found" Sep 09 20:50:22.187904 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.187884 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99981a6272193d33eff6882647ea56fb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal\" (UID: \"99981a6272193d33eff6882647ea56fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal" Sep 09 20:50:22.187973 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.187913 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99981a6272193d33eff6882647ea56fb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal\" (UID: \"99981a6272193d33eff6882647ea56fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal" Sep 09 20:50:22.187973 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.187936 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70b70169d86c3bcf2d0a1e114ed46d9a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-19.ec2.internal\" (UID: \"70b70169d86c3bcf2d0a1e114ed46d9a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-19.ec2.internal" Sep 09 20:50:22.187973 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.187968 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99981a6272193d33eff6882647ea56fb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal\" (UID: \"99981a6272193d33eff6882647ea56fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal" Sep 09 20:50:22.188082 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.187990 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70b70169d86c3bcf2d0a1e114ed46d9a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-19.ec2.internal\" (UID: \"70b70169d86c3bcf2d0a1e114ed46d9a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-19.ec2.internal" Sep 09 20:50:22.188082 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.187996 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99981a6272193d33eff6882647ea56fb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal\" (UID: \"99981a6272193d33eff6882647ea56fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal" Sep 09 20:50:22.281333 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:22.281280 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-19.ec2.internal\" not found" Sep 09 20:50:22.330725 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.330710 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal" Sep 09 20:50:22.336410 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.336395 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-19.ec2.internal" Sep 09 20:50:22.381689 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:22.381661 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-19.ec2.internal\" not found" Sep 09 20:50:22.482225 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:22.482208 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-19.ec2.internal\" not found" Sep 09 20:50:22.582668 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:22.582647 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-19.ec2.internal\" not found" Sep 09 20:50:22.683315 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:22.683285 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-19.ec2.internal\" not found" Sep 09 20:50:22.698774 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.698756 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 20:50:22.724781 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.724766 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 09 20:50:22.725129 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.724844 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 20:50:22.725129 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.724848 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 20:50:22.769080 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.769061 2580 apiserver.go:52] "Watching apiserver" Sep 09 20:50:22.784397 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.784380 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 09 20:50:22.784502 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.784386 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 09 20:50:22.785083 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.784471 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal" Sep 09 20:50:22.785593 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.785568 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-csblg","kube-system/konnectivity-agent-2g5nr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj","openshift-dns/node-resolver-bxfjt","openshift-image-registry/node-ca-gx8lm","openshift-multus/multus-additional-cni-plugins-rwtrs","openshift-network-diagnostics/network-check-target-tbqnj","openshift-ovn-kubernetes/ovnkube-node-tctqv","openshift-cluster-node-tuning-operator/tuned-c5dnk","openshift-multus/multus-95fw4","openshift-multus/network-metrics-daemon-9dfd9","openshift-network-operator/iptables-alerter-ppc2l"] Sep 09 20:50:22.788145 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.788124 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:22.788232 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:22.788213 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csblg" podUID="91f58455-46ed-4617-bfe6-ef79a8fc05f3" Sep 09 20:50:22.790173 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.790155 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2g5nr" Sep 09 20:50:22.792264 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.792245 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 09 20:50:22.792359 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.792323 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 09 20:50:22.792446 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.792430 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" Sep 09 20:50:22.792497 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.792487 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-smpv9\"" Sep 09 20:50:22.794169 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.794153 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 09 20:50:22.794353 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.794341 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 09 20:50:22.794396 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.794348 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 20:45:22 +0000 UTC" deadline="2028-03-17 16:46:34.504704988 +0000 UTC" Sep 09 20:50:22.794396 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.794363 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13315h56m11.710343713s" Sep 09 20:50:22.794526 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.794513 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 09 20:50:22.794575 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.794541 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bxfjt" Sep 09 20:50:22.794621 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.794594 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q7nkk\"" Sep 09 20:50:22.794683 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.794665 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 20:50:22.796821 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.796807 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gx8lm" Sep 09 20:50:22.799011 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.798997 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rwtrs" Sep 09 20:50:22.799241 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.799224 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 20:50:22.799321 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.799313 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-19.ec2.internal" Sep 09 20:50:22.799496 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.799480 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 09 20:50:22.799554 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.799503 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 09 20:50:22.799554 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.799512 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9rnwg\"" Sep 09 20:50:22.799753 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.799741 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 09 20:50:22.799824 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.799810 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7dp22\"" Sep 09 20:50:22.799951 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.799939 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 09 20:50:22.800006 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.799988 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 09 20:50:22.800811 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.800793 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 09 20:50:22.800899 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.800817 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 09 20:50:22.801111 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.801093 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:22.801185 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:22.801161 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbqnj" podUID="b5e2d5ce-182b-46ef-b30d-52e8115c4f54" Sep 09 20:50:22.801458 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.801339 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 09 20:50:22.801458 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.801361 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 09 20:50:22.801458 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.801394 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pp5xn\"" Sep 09 20:50:22.801458 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.801403 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 09 20:50:22.803493 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.803477 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.805320 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.805304 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 09 20:50:22.805419 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.805406 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 09 20:50:22.805646 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.805632 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 09 20:50:22.805646 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.805637 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.805748 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.805674 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rz22j\"" Sep 09 20:50:22.805887 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.805871 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 09 20:50:22.805934 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.805918 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 09 20:50:22.806102 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.806089 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 09 20:50:22.807436 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.807423 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 09 20:50:22.807675 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.807661 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cjztd\"" Sep 09 20:50:22.807781 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.807765 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 09 20:50:22.807833 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.807815 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.808700 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.808685 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 20:50:22.809572 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.809549 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 09 20:50:22.809572 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.809564 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-g5r2f\"" Sep 09 20:50:22.810069 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.810050 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:22.810174 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:22.810112 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dfd9" podUID="f8e8b1d2-6523-403e-bace-b96b38d8449b" Sep 09 20:50:22.812239 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.812215 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal"] Sep 09 20:50:22.812370 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.812265 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-19.ec2.internal"] Sep 09 20:50:22.812370 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.812331 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ppc2l" Sep 09 20:50:22.814272 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.814258 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bnn6j\"" Sep 09 20:50:22.814358 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.814346 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 09 20:50:22.814399 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.814373 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 09 20:50:22.814462 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.814450 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 09 20:50:22.826741 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.826724 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sv4nm" Sep 09 20:50:22.834000 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.833964 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sv4nm" Sep 09 20:50:22.885651 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.885634 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 09 20:50:22.890902 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.890885 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c9d9f144-f418-422d-9356-b682af358e46-sys-fs\") pod \"aws-ebs-csi-driver-node-4fqlj\" (UID: \"c9d9f144-f418-422d-9356-b682af358e46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" Sep 09 20:50:22.890988 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.890907 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-run-ovn\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.890988 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.890923 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.890988 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.890938 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgz9j\" (UniqueName: \"kubernetes.io/projected/e64d1b19-5ae3-4758-8753-e83a736855f0-kube-api-access-fgz9j\") pod \"node-ca-gx8lm\" (UID: \"e64d1b19-5ae3-4758-8753-e83a736855f0\") " pod="openshift-image-registry/node-ca-gx8lm" Sep 09 20:50:22.890988 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.890951 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75366d82-a9b6-4a08-a9cf-a3c9cd38ef85-system-cni-dir\") pod \"multus-additional-cni-plugins-rwtrs\" (UID: \"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85\") " pod="openshift-multus/multus-additional-cni-plugins-rwtrs" Sep 09 20:50:22.891189 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891014 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/428ea8e3-b433-48c4-9330-9f4ace89b6fa-iptables-alerter-script\") pod \"iptables-alerter-ppc2l\" (UID: \"428ea8e3-b433-48c4-9330-9f4ace89b6fa\") " pod="openshift-network-operator/iptables-alerter-ppc2l" Sep 09 20:50:22.891189 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891045 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-var-lib-openvswitch\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.891189 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891079 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/717799e6-8e3b-4a16-88d6-384d950d5385-hosts-file\") pod \"node-resolver-bxfjt\" (UID: \"717799e6-8e3b-4a16-88d6-384d950d5385\") " pod="openshift-dns/node-resolver-bxfjt" Sep 09 20:50:22.891189 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891103 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-etc-sysconfig\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.891189 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891118 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-host-run-multus-certs\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.891189 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891134 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tjbh\" (UniqueName: \"kubernetes.io/projected/b8cbb806-0e1f-40f8-9595-3001333438ea-kube-api-access-8tjbh\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.891189 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891153 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c9d9f144-f418-422d-9356-b682af358e46-device-dir\") pod \"aws-ebs-csi-driver-node-4fqlj\" (UID: \"c9d9f144-f418-422d-9356-b682af358e46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" Sep 09 20:50:22.891189 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891172 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c9d9f144-f418-422d-9356-b682af358e46-etc-selinux\") pod \"aws-ebs-csi-driver-node-4fqlj\" (UID: \"c9d9f144-f418-422d-9356-b682af358e46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" Sep 09 20:50:22.891189 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891187 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-ovn-node-metrics-cert\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.891517 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891203 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-hostroot\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.891517 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891224 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-multus-conf-dir\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.891517 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891260 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-etc-modprobe-d\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.891517 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891284 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-etc-sysctl-d\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.891517 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891322 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-run\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.891517 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891341 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-system-cni-dir\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.891517 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891358 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-host-run-netns\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.891517 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891377 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-cnibin\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.891517 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891396 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-host-slash\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.891517 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891416 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-run-openvswitch\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.891517 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891446 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-log-socket\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.891517 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891460 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-host-cni-bin\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.891517 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891473 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pld2p\" (UniqueName: \"kubernetes.io/projected/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-kube-api-access-pld2p\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.891517 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891489 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75366d82-a9b6-4a08-a9cf-a3c9cd38ef85-cnibin\") pod \"multus-additional-cni-plugins-rwtrs\" (UID: \"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85\") " pod="openshift-multus/multus-additional-cni-plugins-rwtrs" Sep 09 20:50:22.891517 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891514 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75366d82-a9b6-4a08-a9cf-a3c9cd38ef85-cni-binary-copy\") pod \"multus-additional-cni-plugins-rwtrs\" (UID: \"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85\") " pod="openshift-multus/multus-additional-cni-plugins-rwtrs" Sep 09 20:50:22.892106 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891561 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-ovnkube-config\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.892106 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891596 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-host-var-lib-cni-bin\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.892106 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891619 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-host-var-lib-kubelet\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.892106 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891640 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/91f58455-46ed-4617-bfe6-ef79a8fc05f3-kubelet-config\") pod \"global-pull-secret-syncer-csblg\" (UID: \"91f58455-46ed-4617-bfe6-ef79a8fc05f3\") " pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:22.892106 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891664 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/428ea8e3-b433-48c4-9330-9f4ace89b6fa-host-slash\") pod \"iptables-alerter-ppc2l\" (UID: \"428ea8e3-b433-48c4-9330-9f4ace89b6fa\") " pod="openshift-network-operator/iptables-alerter-ppc2l" Sep 09 20:50:22.892106 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891691 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-systemd-units\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.892106 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891719 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-run-systemd\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.892106 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891738 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-etc-kubernetes\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.892106 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891765 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b8cbb806-0e1f-40f8-9595-3001333438ea-cni-binary-copy\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.892106 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891791 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-multus-socket-dir-parent\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.892106 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891827 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e64d1b19-5ae3-4758-8753-e83a736855f0-host\") pod \"node-ca-gx8lm\" (UID: \"e64d1b19-5ae3-4758-8753-e83a736855f0\") " pod="openshift-image-registry/node-ca-gx8lm" Sep 09 20:50:22.892106 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891849 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c859f8c-e68a-4579-966e-8e01d3f66928-tmp\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.892106 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891886 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c9d9f144-f418-422d-9356-b682af358e46-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4fqlj\" (UID: \"c9d9f144-f418-422d-9356-b682af358e46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" Sep 09 20:50:22.892106 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891919 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q27g\" (UniqueName: \"kubernetes.io/projected/717799e6-8e3b-4a16-88d6-384d950d5385-kube-api-access-2q27g\") pod \"node-resolver-bxfjt\" (UID: \"717799e6-8e3b-4a16-88d6-384d950d5385\") " pod="openshift-dns/node-resolver-bxfjt" Sep 09 20:50:22.892106 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891943 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/75366d82-a9b6-4a08-a9cf-a3c9cd38ef85-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rwtrs\" (UID: \"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85\") " pod="openshift-multus/multus-additional-cni-plugins-rwtrs" Sep 09 20:50:22.892106 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.891972 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzjcn\" (UniqueName: \"kubernetes.io/projected/b5e2d5ce-182b-46ef-b30d-52e8115c4f54-kube-api-access-bzjcn\") pod \"network-check-target-tbqnj\" (UID: \"b5e2d5ce-182b-46ef-b30d-52e8115c4f54\") " pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:22.892743 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892000 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-host-cni-netd\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.892743 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892025 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-multus-cni-dir\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.892743 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892064 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75366d82-a9b6-4a08-a9cf-a3c9cd38ef85-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rwtrs\" (UID: \"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85\") " pod="openshift-multus/multus-additional-cni-plugins-rwtrs" Sep 09 20:50:22.892743 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892113 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-etc-systemd\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.892743 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892135 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-sys\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.892743 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892159 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c9d9f144-f418-422d-9356-b682af358e46-socket-dir\") pod \"aws-ebs-csi-driver-node-4fqlj\" (UID: \"c9d9f144-f418-422d-9356-b682af358e46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" Sep 09 20:50:22.892743 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892180 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-etc-openvswitch\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.892743 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892208 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-host\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.892743 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892228 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e64d1b19-5ae3-4758-8753-e83a736855f0-serviceca\") pod \"node-ca-gx8lm\" (UID: \"e64d1b19-5ae3-4758-8753-e83a736855f0\") " pod="openshift-image-registry/node-ca-gx8lm" Sep 09 20:50:22.892743 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892248 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs\") pod \"network-metrics-daemon-9dfd9\" (UID: \"f8e8b1d2-6523-403e-bace-b96b38d8449b\") " pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:22.892743 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892271 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kzsx\" (UniqueName: \"kubernetes.io/projected/f8e8b1d2-6523-403e-bace-b96b38d8449b-kube-api-access-5kzsx\") pod \"network-metrics-daemon-9dfd9\" (UID: \"f8e8b1d2-6523-403e-bace-b96b38d8449b\") " pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:22.892743 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/91f58455-46ed-4617-bfe6-ef79a8fc05f3-dbus\") pod \"global-pull-secret-syncer-csblg\" (UID: \"91f58455-46ed-4617-bfe6-ef79a8fc05f3\") " pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:22.892743 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892361 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/717799e6-8e3b-4a16-88d6-384d950d5385-tmp-dir\") pod \"node-resolver-bxfjt\" (UID: \"717799e6-8e3b-4a16-88d6-384d950d5385\") " pod="openshift-dns/node-resolver-bxfjt" Sep 09 20:50:22.892743 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892408 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-node-log\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.892743 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892444 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-env-overrides\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.892743 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892473 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f5167c4c-e0f9-496f-b62f-78764ba3cbd7-konnectivity-ca\") pod \"konnectivity-agent-2g5nr\" (UID: \"f5167c4c-e0f9-496f-b62f-78764ba3cbd7\") " pod="kube-system/konnectivity-agent-2g5nr" Sep 09 20:50:22.892743 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892494 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-var-lib-kubelet\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.893422 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892520 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c859f8c-e68a-4579-966e-8e01d3f66928-etc-tuned\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.893422 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892543 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdg4z\" (UniqueName: \"kubernetes.io/projected/428ea8e3-b433-48c4-9330-9f4ace89b6fa-kube-api-access-cdg4z\") pod \"iptables-alerter-ppc2l\" (UID: \"428ea8e3-b433-48c4-9330-9f4ace89b6fa\") " pod="openshift-network-operator/iptables-alerter-ppc2l" Sep 09 20:50:22.893422 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892566 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl7cr\" (UniqueName: \"kubernetes.io/projected/c9d9f144-f418-422d-9356-b682af358e46-kube-api-access-nl7cr\") pod \"aws-ebs-csi-driver-node-4fqlj\" (UID: \"c9d9f144-f418-422d-9356-b682af358e46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" Sep 09 20:50:22.893422 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892587 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-host-kubelet\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.893422 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892609 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-host-run-netns\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.893422 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892639 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-ovnkube-script-lib\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.893422 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892662 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b8cbb806-0e1f-40f8-9595-3001333438ea-multus-daemon-config\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.893422 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892735 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-etc-kubernetes\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.893422 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892766 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8k7j\" (UniqueName: \"kubernetes.io/projected/9c859f8c-e68a-4579-966e-8e01d3f66928-kube-api-access-c8k7j\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.893422 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892789 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c9d9f144-f418-422d-9356-b682af358e46-registration-dir\") pod \"aws-ebs-csi-driver-node-4fqlj\" (UID: \"c9d9f144-f418-422d-9356-b682af358e46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" Sep 09 20:50:22.893422 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892815 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f5167c4c-e0f9-496f-b62f-78764ba3cbd7-agent-certs\") pod \"konnectivity-agent-2g5nr\" (UID: \"f5167c4c-e0f9-496f-b62f-78764ba3cbd7\") " pod="kube-system/konnectivity-agent-2g5nr" Sep 09 20:50:22.893422 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892855 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-host-run-k8s-cni-cncf-io\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.893422 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892879 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-host-var-lib-cni-multus\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.893422 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892903 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt84s\" (UniqueName: \"kubernetes.io/projected/75366d82-a9b6-4a08-a9cf-a3c9cd38ef85-kube-api-access-xt84s\") pod \"multus-additional-cni-plugins-rwtrs\" (UID: \"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85\") " pod="openshift-multus/multus-additional-cni-plugins-rwtrs" Sep 09 20:50:22.893422 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892925 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-etc-sysctl-conf\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.893422 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892947 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-host-run-ovn-kubernetes\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.893980 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.892993 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91f58455-46ed-4617-bfe6-ef79a8fc05f3-original-pull-secret\") pod \"global-pull-secret-syncer-csblg\" (UID: \"91f58455-46ed-4617-bfe6-ef79a8fc05f3\") " pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:22.893980 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.893023 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/75366d82-a9b6-4a08-a9cf-a3c9cd38ef85-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rwtrs\" (UID: \"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85\") " pod="openshift-multus/multus-additional-cni-plugins-rwtrs" Sep 09 20:50:22.893980 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.893045 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-lib-modules\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.921933 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.921913 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 20:50:22.993594 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.993571 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f5167c4c-e0f9-496f-b62f-78764ba3cbd7-konnectivity-ca\") pod \"konnectivity-agent-2g5nr\" (UID: \"f5167c4c-e0f9-496f-b62f-78764ba3cbd7\") " pod="kube-system/konnectivity-agent-2g5nr" Sep 09 20:50:22.993594 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.993597 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-var-lib-kubelet\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.993736 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.993612 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c859f8c-e68a-4579-966e-8e01d3f66928-etc-tuned\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.993736 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.993626 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdg4z\" (UniqueName: \"kubernetes.io/projected/428ea8e3-b433-48c4-9330-9f4ace89b6fa-kube-api-access-cdg4z\") pod \"iptables-alerter-ppc2l\" (UID: \"428ea8e3-b433-48c4-9330-9f4ace89b6fa\") " pod="openshift-network-operator/iptables-alerter-ppc2l" Sep 09 20:50:22.993736 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.993692 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-var-lib-kubelet\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.993736 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.993695 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl7cr\" (UniqueName: \"kubernetes.io/projected/c9d9f144-f418-422d-9356-b682af358e46-kube-api-access-nl7cr\") pod \"aws-ebs-csi-driver-node-4fqlj\" (UID: \"c9d9f144-f418-422d-9356-b682af358e46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" Sep 09 20:50:22.993937 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.993750 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-host-kubelet\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.993937 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.993773 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-host-run-netns\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.993937 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.993795 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-ovnkube-script-lib\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.993937 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.993820 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b8cbb806-0e1f-40f8-9595-3001333438ea-multus-daemon-config\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.993937 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.993839 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-etc-kubernetes\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.993937 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.993853 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-host-kubelet\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.993937 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.993860 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8k7j\" (UniqueName: \"kubernetes.io/projected/9c859f8c-e68a-4579-966e-8e01d3f66928-kube-api-access-c8k7j\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.993937 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.993908 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c9d9f144-f418-422d-9356-b682af358e46-registration-dir\") pod \"aws-ebs-csi-driver-node-4fqlj\" (UID: \"c9d9f144-f418-422d-9356-b682af358e46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" Sep 09 20:50:22.993937 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.993933 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f5167c4c-e0f9-496f-b62f-78764ba3cbd7-agent-certs\") pod \"konnectivity-agent-2g5nr\" (UID: \"f5167c4c-e0f9-496f-b62f-78764ba3cbd7\") " pod="kube-system/konnectivity-agent-2g5nr" Sep 09 20:50:22.994373 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.993956 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-host-run-k8s-cni-cncf-io\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.994373 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.993978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-host-var-lib-cni-multus\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.994373 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.993964 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 09 20:50:22.994373 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994003 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt84s\" (UniqueName: \"kubernetes.io/projected/75366d82-a9b6-4a08-a9cf-a3c9cd38ef85-kube-api-access-xt84s\") pod \"multus-additional-cni-plugins-rwtrs\" (UID: \"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85\") " pod="openshift-multus/multus-additional-cni-plugins-rwtrs" Sep 09 20:50:22.994373 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994007 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c9d9f144-f418-422d-9356-b682af358e46-registration-dir\") pod \"aws-ebs-csi-driver-node-4fqlj\" (UID: \"c9d9f144-f418-422d-9356-b682af358e46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" Sep 09 20:50:22.994373 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994030 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-etc-sysctl-conf\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.994373 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994069 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-host-run-k8s-cni-cncf-io\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.994373 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994098 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-host-run-ovn-kubernetes\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.994373 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994102 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-etc-kubernetes\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.994373 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994119 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91f58455-46ed-4617-bfe6-ef79a8fc05f3-original-pull-secret\") pod \"global-pull-secret-syncer-csblg\" (UID: \"91f58455-46ed-4617-bfe6-ef79a8fc05f3\") " pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:22.994373 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994143 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-etc-sysctl-conf\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.994373 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994177 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-host-run-ovn-kubernetes\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.994373 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994175 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f5167c4c-e0f9-496f-b62f-78764ba3cbd7-konnectivity-ca\") pod \"konnectivity-agent-2g5nr\" (UID: \"f5167c4c-e0f9-496f-b62f-78764ba3cbd7\") " pod="kube-system/konnectivity-agent-2g5nr" Sep 09 20:50:22.994373 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:22.994334 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 20:50:22.994951 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:22.994404 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91f58455-46ed-4617-bfe6-ef79a8fc05f3-original-pull-secret podName:91f58455-46ed-4617-bfe6-ef79a8fc05f3 nodeName:}" failed. No retries permitted until 2026-09-09 20:50:23.494385022 +0000 UTC m=+2.100627718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91f58455-46ed-4617-bfe6-ef79a8fc05f3-original-pull-secret") pod "global-pull-secret-syncer-csblg" (UID: "91f58455-46ed-4617-bfe6-ef79a8fc05f3") : object "kube-system"/"original-pull-secret" not registered Sep 09 20:50:22.994951 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994415 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-ovnkube-script-lib\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.994951 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994143 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/75366d82-a9b6-4a08-a9cf-a3c9cd38ef85-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rwtrs\" (UID: \"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85\") " pod="openshift-multus/multus-additional-cni-plugins-rwtrs" Sep 09 20:50:22.994951 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994416 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-host-var-lib-cni-multus\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.994951 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994456 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-lib-modules\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.994951 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994482 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c9d9f144-f418-422d-9356-b682af358e46-sys-fs\") pod \"aws-ebs-csi-driver-node-4fqlj\" (UID: \"c9d9f144-f418-422d-9356-b682af358e46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" Sep 09 20:50:22.994951 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994504 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-run-ovn\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.994951 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994511 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b8cbb806-0e1f-40f8-9595-3001333438ea-multus-daemon-config\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.994951 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994504 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-host-run-netns\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.994951 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994527 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.994951 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994573 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.994951 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994590 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c9d9f144-f418-422d-9356-b682af358e46-sys-fs\") pod \"aws-ebs-csi-driver-node-4fqlj\" (UID: \"c9d9f144-f418-422d-9356-b682af358e46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" Sep 09 20:50:22.994951 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994610 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgz9j\" (UniqueName: \"kubernetes.io/projected/e64d1b19-5ae3-4758-8753-e83a736855f0-kube-api-access-fgz9j\") pod \"node-ca-gx8lm\" (UID: \"e64d1b19-5ae3-4758-8753-e83a736855f0\") " pod="openshift-image-registry/node-ca-gx8lm" Sep 09 20:50:22.994951 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994645 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-lib-modules\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.994951 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994655 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75366d82-a9b6-4a08-a9cf-a3c9cd38ef85-system-cni-dir\") pod \"multus-additional-cni-plugins-rwtrs\" (UID: \"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85\") " pod="openshift-multus/multus-additional-cni-plugins-rwtrs" Sep 09 20:50:22.994951 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994650 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/75366d82-a9b6-4a08-a9cf-a3c9cd38ef85-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rwtrs\" (UID: \"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85\") " pod="openshift-multus/multus-additional-cni-plugins-rwtrs" Sep 09 20:50:22.995832 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994680 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/428ea8e3-b433-48c4-9330-9f4ace89b6fa-iptables-alerter-script\") pod \"iptables-alerter-ppc2l\" (UID: \"428ea8e3-b433-48c4-9330-9f4ace89b6fa\") " pod="openshift-network-operator/iptables-alerter-ppc2l" Sep 09 20:50:22.995832 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994701 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-var-lib-openvswitch\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.995832 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994706 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-run-ovn\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.995832 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994709 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75366d82-a9b6-4a08-a9cf-a3c9cd38ef85-system-cni-dir\") pod \"multus-additional-cni-plugins-rwtrs\" (UID: \"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85\") " pod="openshift-multus/multus-additional-cni-plugins-rwtrs" Sep 09 20:50:22.995832 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994721 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/717799e6-8e3b-4a16-88d6-384d950d5385-hosts-file\") pod \"node-resolver-bxfjt\" (UID: \"717799e6-8e3b-4a16-88d6-384d950d5385\") " pod="openshift-dns/node-resolver-bxfjt" Sep 09 20:50:22.995832 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994744 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-etc-sysconfig\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.995832 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994764 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-host-run-multus-certs\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.995832 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994783 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tjbh\" (UniqueName: \"kubernetes.io/projected/b8cbb806-0e1f-40f8-9595-3001333438ea-kube-api-access-8tjbh\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.995832 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994803 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c9d9f144-f418-422d-9356-b682af358e46-device-dir\") pod \"aws-ebs-csi-driver-node-4fqlj\" (UID: \"c9d9f144-f418-422d-9356-b682af358e46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" Sep 09 20:50:22.995832 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994821 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c9d9f144-f418-422d-9356-b682af358e46-etc-selinux\") pod \"aws-ebs-csi-driver-node-4fqlj\" (UID: \"c9d9f144-f418-422d-9356-b682af358e46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" Sep 09 20:50:22.995832 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994825 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-etc-sysconfig\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.995832 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994752 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-var-lib-openvswitch\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.995832 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994840 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-ovn-node-metrics-cert\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.995832 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994849 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/717799e6-8e3b-4a16-88d6-384d950d5385-hosts-file\") pod \"node-resolver-bxfjt\" (UID: \"717799e6-8e3b-4a16-88d6-384d950d5385\") " pod="openshift-dns/node-resolver-bxfjt" Sep 09 20:50:22.995832 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994935 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c9d9f144-f418-422d-9356-b682af358e46-device-dir\") pod \"aws-ebs-csi-driver-node-4fqlj\" (UID: \"c9d9f144-f418-422d-9356-b682af358e46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" Sep 09 20:50:22.995832 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.994988 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c9d9f144-f418-422d-9356-b682af358e46-etc-selinux\") pod \"aws-ebs-csi-driver-node-4fqlj\" (UID: \"c9d9f144-f418-422d-9356-b682af358e46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" Sep 09 20:50:22.995832 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995011 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-host-run-multus-certs\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.996657 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995049 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-hostroot\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.996657 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995070 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-multus-conf-dir\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.996657 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995083 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-etc-modprobe-d\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.996657 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995106 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-etc-sysctl-d\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.996657 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995123 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-run\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.996657 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995123 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-hostroot\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.996657 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995143 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-system-cni-dir\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.996657 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995160 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-host-run-netns\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.996657 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995191 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-etc-modprobe-d\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.996657 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995213 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/428ea8e3-b433-48c4-9330-9f4ace89b6fa-iptables-alerter-script\") pod \"iptables-alerter-ppc2l\" (UID: \"428ea8e3-b433-48c4-9330-9f4ace89b6fa\") " pod="openshift-network-operator/iptables-alerter-ppc2l" Sep 09 20:50:22.996657 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995226 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-cnibin\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.996657 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995228 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-host-run-netns\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.996657 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995260 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-run\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.996657 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995263 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-etc-sysctl-d\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.996657 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995279 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-system-cni-dir\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.996657 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995314 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-host-slash\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.996657 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995399 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-run-openvswitch\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.996657 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995414 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-log-socket\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.997458 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995347 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-host-slash\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.997458 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995342 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-cnibin\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.997458 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995467 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-host-cni-bin\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.997458 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995431 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-multus-conf-dir\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.997458 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995484 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pld2p\" (UniqueName: \"kubernetes.io/projected/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-kube-api-access-pld2p\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.997458 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995512 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-run-openvswitch\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.997458 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995537 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-host-cni-bin\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.997458 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995547 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75366d82-a9b6-4a08-a9cf-a3c9cd38ef85-cnibin\") pod \"multus-additional-cni-plugins-rwtrs\" (UID: \"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85\") " pod="openshift-multus/multus-additional-cni-plugins-rwtrs" Sep 09 20:50:22.997458 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995578 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75366d82-a9b6-4a08-a9cf-a3c9cd38ef85-cnibin\") pod \"multus-additional-cni-plugins-rwtrs\" (UID: \"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85\") " pod="openshift-multus/multus-additional-cni-plugins-rwtrs" Sep 09 20:50:22.997458 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995584 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75366d82-a9b6-4a08-a9cf-a3c9cd38ef85-cni-binary-copy\") pod \"multus-additional-cni-plugins-rwtrs\" (UID: \"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85\") " pod="openshift-multus/multus-additional-cni-plugins-rwtrs" Sep 09 20:50:22.997458 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995611 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-ovnkube-config\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.997458 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995633 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-host-var-lib-cni-bin\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.997458 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995655 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-host-var-lib-kubelet\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.997458 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995676 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-log-socket\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.997458 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995707 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-host-var-lib-cni-bin\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.997458 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995717 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-host-var-lib-kubelet\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.997458 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995680 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/91f58455-46ed-4617-bfe6-ef79a8fc05f3-kubelet-config\") pod \"global-pull-secret-syncer-csblg\" (UID: \"91f58455-46ed-4617-bfe6-ef79a8fc05f3\") " pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:22.997458 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995733 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/91f58455-46ed-4617-bfe6-ef79a8fc05f3-kubelet-config\") pod \"global-pull-secret-syncer-csblg\" (UID: \"91f58455-46ed-4617-bfe6-ef79a8fc05f3\") " pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:22.998071 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995753 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/428ea8e3-b433-48c4-9330-9f4ace89b6fa-host-slash\") pod \"iptables-alerter-ppc2l\" (UID: \"428ea8e3-b433-48c4-9330-9f4ace89b6fa\") " pod="openshift-network-operator/iptables-alerter-ppc2l" Sep 09 20:50:22.998071 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995778 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-systemd-units\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.998071 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995800 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-run-systemd\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.998071 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995823 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-etc-kubernetes\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.998071 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995846 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b8cbb806-0e1f-40f8-9595-3001333438ea-cni-binary-copy\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.998071 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995854 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-systemd-units\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.998071 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995868 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-multus-socket-dir-parent\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.998071 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995895 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e64d1b19-5ae3-4758-8753-e83a736855f0-host\") pod \"node-ca-gx8lm\" (UID: \"e64d1b19-5ae3-4758-8753-e83a736855f0\") " pod="openshift-image-registry/node-ca-gx8lm" Sep 09 20:50:22.998071 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995916 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c859f8c-e68a-4579-966e-8e01d3f66928-tmp\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.998071 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995944 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c9d9f144-f418-422d-9356-b682af358e46-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4fqlj\" (UID: \"c9d9f144-f418-422d-9356-b682af358e46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" Sep 09 20:50:22.998071 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995948 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-run-systemd\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.998071 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995895 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/428ea8e3-b433-48c4-9330-9f4ace89b6fa-host-slash\") pod \"iptables-alerter-ppc2l\" (UID: \"428ea8e3-b433-48c4-9330-9f4ace89b6fa\") " pod="openshift-network-operator/iptables-alerter-ppc2l" Sep 09 20:50:22.998071 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995969 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q27g\" (UniqueName: \"kubernetes.io/projected/717799e6-8e3b-4a16-88d6-384d950d5385-kube-api-access-2q27g\") pod \"node-resolver-bxfjt\" (UID: \"717799e6-8e3b-4a16-88d6-384d950d5385\") " pod="openshift-dns/node-resolver-bxfjt" Sep 09 20:50:22.998071 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.995992 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/75366d82-a9b6-4a08-a9cf-a3c9cd38ef85-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rwtrs\" (UID: \"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85\") " pod="openshift-multus/multus-additional-cni-plugins-rwtrs" Sep 09 20:50:22.998071 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996010 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75366d82-a9b6-4a08-a9cf-a3c9cd38ef85-cni-binary-copy\") pod \"multus-additional-cni-plugins-rwtrs\" (UID: \"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85\") " pod="openshift-multus/multus-additional-cni-plugins-rwtrs" Sep 09 20:50:22.998071 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996015 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzjcn\" (UniqueName: \"kubernetes.io/projected/b5e2d5ce-182b-46ef-b30d-52e8115c4f54-kube-api-access-bzjcn\") pod \"network-check-target-tbqnj\" (UID: \"b5e2d5ce-182b-46ef-b30d-52e8115c4f54\") " pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:22.998071 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996061 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-host-cni-netd\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.998547 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996083 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-multus-cni-dir\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.998547 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996109 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75366d82-a9b6-4a08-a9cf-a3c9cd38ef85-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rwtrs\" (UID: \"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85\") " pod="openshift-multus/multus-additional-cni-plugins-rwtrs" Sep 09 20:50:22.998547 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996126 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-ovnkube-config\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.998547 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996134 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-etc-systemd\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.998547 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-sys\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.998547 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996178 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c9d9f144-f418-422d-9356-b682af358e46-socket-dir\") pod \"aws-ebs-csi-driver-node-4fqlj\" (UID: \"c9d9f144-f418-422d-9356-b682af358e46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" Sep 09 20:50:22.998547 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996183 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c9d9f144-f418-422d-9356-b682af358e46-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4fqlj\" (UID: \"c9d9f144-f418-422d-9356-b682af358e46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" Sep 09 20:50:22.998547 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996200 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-etc-openvswitch\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.998547 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996227 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-etc-kubernetes\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.998547 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996244 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-etc-openvswitch\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.998547 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996314 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-host-cni-netd\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.998547 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996367 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-multus-cni-dir\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.998547 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996443 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b8cbb806-0e1f-40f8-9595-3001333438ea-cni-binary-copy\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.998547 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996514 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75366d82-a9b6-4a08-a9cf-a3c9cd38ef85-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rwtrs\" (UID: \"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85\") " pod="openshift-multus/multus-additional-cni-plugins-rwtrs" Sep 09 20:50:22.998547 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996538 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b8cbb806-0e1f-40f8-9595-3001333438ea-multus-socket-dir-parent\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:22.998547 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996567 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-etc-systemd\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.998547 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996581 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e64d1b19-5ae3-4758-8753-e83a736855f0-host\") pod \"node-ca-gx8lm\" (UID: \"e64d1b19-5ae3-4758-8753-e83a736855f0\") " pod="openshift-image-registry/node-ca-gx8lm" Sep 09 20:50:22.998547 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996613 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-host\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.999072 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996640 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e64d1b19-5ae3-4758-8753-e83a736855f0-serviceca\") pod \"node-ca-gx8lm\" (UID: \"e64d1b19-5ae3-4758-8753-e83a736855f0\") " pod="openshift-image-registry/node-ca-gx8lm" Sep 09 20:50:22.999072 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996649 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-host\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.999072 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996615 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c859f8c-e68a-4579-966e-8e01d3f66928-sys\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.999072 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996662 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs\") pod \"network-metrics-daemon-9dfd9\" (UID: \"f8e8b1d2-6523-403e-bace-b96b38d8449b\") " pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:22.999072 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996693 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c9d9f144-f418-422d-9356-b682af358e46-socket-dir\") pod \"aws-ebs-csi-driver-node-4fqlj\" (UID: \"c9d9f144-f418-422d-9356-b682af358e46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" Sep 09 20:50:22.999072 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996696 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kzsx\" (UniqueName: \"kubernetes.io/projected/f8e8b1d2-6523-403e-bace-b96b38d8449b-kube-api-access-5kzsx\") pod \"network-metrics-daemon-9dfd9\" (UID: \"f8e8b1d2-6523-403e-bace-b96b38d8449b\") " pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:22.999072 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:22.996723 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:50:22.999072 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996766 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/91f58455-46ed-4617-bfe6-ef79a8fc05f3-dbus\") pod \"global-pull-secret-syncer-csblg\" (UID: \"91f58455-46ed-4617-bfe6-ef79a8fc05f3\") " pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:22.999072 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996803 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/75366d82-a9b6-4a08-a9cf-a3c9cd38ef85-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rwtrs\" (UID: \"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85\") " pod="openshift-multus/multus-additional-cni-plugins-rwtrs" Sep 09 20:50:22.999072 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:22.996822 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs podName:f8e8b1d2-6523-403e-bace-b96b38d8449b nodeName:}" failed. No retries permitted until 2026-09-09 20:50:23.496805805 +0000 UTC m=+2.103048496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs") pod "network-metrics-daemon-9dfd9" (UID: "f8e8b1d2-6523-403e-bace-b96b38d8449b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:50:22.999072 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996725 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/91f58455-46ed-4617-bfe6-ef79a8fc05f3-dbus\") pod \"global-pull-secret-syncer-csblg\" (UID: \"91f58455-46ed-4617-bfe6-ef79a8fc05f3\") " pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:22.999072 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996855 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/717799e6-8e3b-4a16-88d6-384d950d5385-tmp-dir\") pod \"node-resolver-bxfjt\" (UID: \"717799e6-8e3b-4a16-88d6-384d950d5385\") " pod="openshift-dns/node-resolver-bxfjt" Sep 09 20:50:22.999072 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996878 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-node-log\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.999072 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.996901 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-env-overrides\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.999072 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.997013 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-node-log\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.999072 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.997160 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/717799e6-8e3b-4a16-88d6-384d950d5385-tmp-dir\") pod \"node-resolver-bxfjt\" (UID: \"717799e6-8e3b-4a16-88d6-384d950d5385\") " pod="openshift-dns/node-resolver-bxfjt" Sep 09 20:50:22.999072 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.997176 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e64d1b19-5ae3-4758-8753-e83a736855f0-serviceca\") pod \"node-ca-gx8lm\" (UID: \"e64d1b19-5ae3-4758-8753-e83a736855f0\") " pod="openshift-image-registry/node-ca-gx8lm" Sep 09 20:50:22.999544 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.997405 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-env-overrides\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.999544 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.997814 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c859f8c-e68a-4579-966e-8e01d3f66928-etc-tuned\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:22.999544 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.997893 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f5167c4c-e0f9-496f-b62f-78764ba3cbd7-agent-certs\") pod \"konnectivity-agent-2g5nr\" (UID: \"f5167c4c-e0f9-496f-b62f-78764ba3cbd7\") " pod="kube-system/konnectivity-agent-2g5nr" Sep 09 20:50:22.999544 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.997929 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-ovn-node-metrics-cert\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:22.999544 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:22.998385 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c859f8c-e68a-4579-966e-8e01d3f66928-tmp\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:23.001390 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:23.001362 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl7cr\" (UniqueName: \"kubernetes.io/projected/c9d9f144-f418-422d-9356-b682af358e46-kube-api-access-nl7cr\") pod \"aws-ebs-csi-driver-node-4fqlj\" (UID: \"c9d9f144-f418-422d-9356-b682af358e46\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" Sep 09 20:50:23.002177 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:23.002151 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdg4z\" (UniqueName: \"kubernetes.io/projected/428ea8e3-b433-48c4-9330-9f4ace89b6fa-kube-api-access-cdg4z\") pod \"iptables-alerter-ppc2l\" (UID: \"428ea8e3-b433-48c4-9330-9f4ace89b6fa\") " pod="openshift-network-operator/iptables-alerter-ppc2l" Sep 09 20:50:23.002353 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:23.002334 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt84s\" (UniqueName: \"kubernetes.io/projected/75366d82-a9b6-4a08-a9cf-a3c9cd38ef85-kube-api-access-xt84s\") pod \"multus-additional-cni-plugins-rwtrs\" (UID: \"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85\") " pod="openshift-multus/multus-additional-cni-plugins-rwtrs" Sep 09 20:50:23.002483 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:23.002470 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8k7j\" (UniqueName: \"kubernetes.io/projected/9c859f8c-e68a-4579-966e-8e01d3f66928-kube-api-access-c8k7j\") pod \"tuned-c5dnk\" (UID: \"9c859f8c-e68a-4579-966e-8e01d3f66928\") " pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:23.007830 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:23.007806 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgz9j\" (UniqueName: \"kubernetes.io/projected/e64d1b19-5ae3-4758-8753-e83a736855f0-kube-api-access-fgz9j\") pod \"node-ca-gx8lm\" (UID: \"e64d1b19-5ae3-4758-8753-e83a736855f0\") " pod="openshift-image-registry/node-ca-gx8lm" Sep 09 20:50:23.009380 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:23.009356 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 20:50:23.009380 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:23.009378 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 20:50:23.009478 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:23.009390 2580 projected.go:196] Error preparing data for projected volume kube-api-access-bzjcn for pod openshift-network-diagnostics/network-check-target-tbqnj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:50:23.009478 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:23.009458 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5e2d5ce-182b-46ef-b30d-52e8115c4f54-kube-api-access-bzjcn podName:b5e2d5ce-182b-46ef-b30d-52e8115c4f54 nodeName:}" failed. No retries permitted until 2026-09-09 20:50:23.509444138 +0000 UTC m=+2.115686825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bzjcn" (UniqueName: "kubernetes.io/projected/b5e2d5ce-182b-46ef-b30d-52e8115c4f54-kube-api-access-bzjcn") pod "network-check-target-tbqnj" (UID: "b5e2d5ce-182b-46ef-b30d-52e8115c4f54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:50:23.011040 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:23.011018 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tjbh\" (UniqueName: \"kubernetes.io/projected/b8cbb806-0e1f-40f8-9595-3001333438ea-kube-api-access-8tjbh\") pod \"multus-95fw4\" (UID: \"b8cbb806-0e1f-40f8-9595-3001333438ea\") " pod="openshift-multus/multus-95fw4" Sep 09 20:50:23.011279 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:23.011264 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q27g\" (UniqueName: \"kubernetes.io/projected/717799e6-8e3b-4a16-88d6-384d950d5385-kube-api-access-2q27g\") pod \"node-resolver-bxfjt\" (UID: \"717799e6-8e3b-4a16-88d6-384d950d5385\") " pod="openshift-dns/node-resolver-bxfjt" Sep 09 20:50:23.011474 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:23.011460 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kzsx\" (UniqueName: \"kubernetes.io/projected/f8e8b1d2-6523-403e-bace-b96b38d8449b-kube-api-access-5kzsx\") pod \"network-metrics-daemon-9dfd9\" (UID: \"f8e8b1d2-6523-403e-bace-b96b38d8449b\") " pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:23.011575 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:23.011561 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pld2p\" (UniqueName: \"kubernetes.io/projected/ddb3d74a-a1bb-4bda-8d0d-89320e50f357-kube-api-access-pld2p\") pod \"ovnkube-node-tctqv\" (UID: \"ddb3d74a-a1bb-4bda-8d0d-89320e50f357\") " pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:23.038269 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:23.038239 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99981a6272193d33eff6882647ea56fb.slice/crio-96d4501bbcb30b7ab4a8580872d3ab4fa49f52d81dfbd93bf836c4dc5b74c537 WatchSource:0}: Error finding container 96d4501bbcb30b7ab4a8580872d3ab4fa49f52d81dfbd93bf836c4dc5b74c537: Status 404 returned error can't find the container with id 96d4501bbcb30b7ab4a8580872d3ab4fa49f52d81dfbd93bf836c4dc5b74c537 Sep 09 20:50:23.038529 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:23.038510 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b70169d86c3bcf2d0a1e114ed46d9a.slice/crio-aab547ce524fe1606845216935c44d4477cf262d37b779122bb2c34f38b1bc6c WatchSource:0}: Error finding container aab547ce524fe1606845216935c44d4477cf262d37b779122bb2c34f38b1bc6c: Status 404 returned error can't find the container with id aab547ce524fe1606845216935c44d4477cf262d37b779122bb2c34f38b1bc6c Sep 09 20:50:23.043342 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:23.043324 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 20:50:23.115845 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:23.115802 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2g5nr" Sep 09 20:50:23.124308 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:23.124272 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5167c4c_e0f9_496f_b62f_78764ba3cbd7.slice/crio-217fb274b4178cfdcad3bf08b2498ba567f7e2fd5a47555d0606c4fd00ef98b4 WatchSource:0}: Error finding container 217fb274b4178cfdcad3bf08b2498ba567f7e2fd5a47555d0606c4fd00ef98b4: Status 404 returned error can't find the container with id 217fb274b4178cfdcad3bf08b2498ba567f7e2fd5a47555d0606c4fd00ef98b4 Sep 09 20:50:23.136230 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:23.136215 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" Sep 09 20:50:23.143653 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:23.143626 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d9f144_f418_422d_9356_b682af358e46.slice/crio-c0643926ad444cbae7a3077f2c8193ce4e2efe7a1906a0bc54e8a014058e4a84 WatchSource:0}: Error finding container c0643926ad444cbae7a3077f2c8193ce4e2efe7a1906a0bc54e8a014058e4a84: Status 404 returned error can't find the container with id c0643926ad444cbae7a3077f2c8193ce4e2efe7a1906a0bc54e8a014058e4a84 Sep 09 20:50:23.153919 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:23.153903 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 20:50:23.155483 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:23.155471 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bxfjt" Sep 09 20:50:23.163494 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:23.163473 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717799e6_8e3b_4a16_88d6_384d950d5385.slice/crio-0434e985a6a45e98375c01773e358fd2ea0e06ac2eeb3f232d279ad2be451d70 WatchSource:0}: Error finding container 0434e985a6a45e98375c01773e358fd2ea0e06ac2eeb3f232d279ad2be451d70: Status 404 returned error can't find the container with id 0434e985a6a45e98375c01773e358fd2ea0e06ac2eeb3f232d279ad2be451d70 Sep 09 20:50:23.172354 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:23.172341 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gx8lm" Sep 09 20:50:23.179640 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:23.179624 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64d1b19_5ae3_4758_8753_e83a736855f0.slice/crio-878718beecc9be0912cf6febe13c53a867c9069d8c756613ca7b3fd74f36b7f7 WatchSource:0}: Error finding container 878718beecc9be0912cf6febe13c53a867c9069d8c756613ca7b3fd74f36b7f7: Status 404 returned error can't find the container with id 878718beecc9be0912cf6febe13c53a867c9069d8c756613ca7b3fd74f36b7f7 Sep 09 20:50:23.191368 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:23.191352 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rwtrs" Sep 09 20:50:23.198049 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:23.198023 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:23.198666 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:23.198648 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75366d82_a9b6_4a08_a9cf_a3c9cd38ef85.slice/crio-147f9ccc144b42be2325abb4113e5447ac93f185f2020ab55c95072753eec499 WatchSource:0}: Error finding container 147f9ccc144b42be2325abb4113e5447ac93f185f2020ab55c95072753eec499: Status 404 returned error can't find the container with id 147f9ccc144b42be2325abb4113e5447ac93f185f2020ab55c95072753eec499 Sep 09 20:50:23.205141 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:23.205125 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb3d74a_a1bb_4bda_8d0d_89320e50f357.slice/crio-cf3514026a11bdf7d07ca7da3b5d91647fbce5f62ada60e412046f1709f051a3 WatchSource:0}: Error finding container cf3514026a11bdf7d07ca7da3b5d91647fbce5f62ada60e412046f1709f051a3: Status 404 returned error can't find the container with id cf3514026a11bdf7d07ca7da3b5d91647fbce5f62ada60e412046f1709f051a3 Sep 09 20:50:23.215053 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:23.215040 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" Sep 09 20:50:23.222572 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:23.222556 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c859f8c_e68a_4579_966e_8e01d3f66928.slice/crio-e62a0a74e798b45d443d504e090b9e1f40ae9bdd098f19e4bd9f5d656bf39791 WatchSource:0}: Error finding container e62a0a74e798b45d443d504e090b9e1f40ae9bdd098f19e4bd9f5d656bf39791: Status 404 returned error can't find the container with id e62a0a74e798b45d443d504e090b9e1f40ae9bdd098f19e4bd9f5d656bf39791 Sep 09 20:50:23.226124 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:23.226103 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-95fw4" Sep 09 20:50:23.230941 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:23.230925 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ppc2l" Sep 09 20:50:23.234386 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:23.234365 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8cbb806_0e1f_40f8_9595_3001333438ea.slice/crio-e5f71b9bdf43e3f4da75ab8e5b69404de677f16683e19ed0a0255ed0bbe1db0b WatchSource:0}: Error finding container e5f71b9bdf43e3f4da75ab8e5b69404de677f16683e19ed0a0255ed0bbe1db0b: Status 404 returned error can't find the container with id e5f71b9bdf43e3f4da75ab8e5b69404de677f16683e19ed0a0255ed0bbe1db0b Sep 09 20:50:23.239817 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:23.239800 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428ea8e3_b433_48c4_9330_9f4ace89b6fa.slice/crio-53150155988d899a061614948c83a48f5794c1d81b6a0cc6ed9accc2c0602e6e WatchSource:0}: Error finding container 53150155988d899a061614948c83a48f5794c1d81b6a0cc6ed9accc2c0602e6e: Status 404 returned error can't find the container with id 53150155988d899a061614948c83a48f5794c1d81b6a0cc6ed9accc2c0602e6e Sep 09 20:50:23.241253 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:23.241233 2580 kuberuntime_manager.go:1459] "Unhandled Error" err="container iptables-alerter start failed in pod iptables-alerter-ppc2l_openshift-network-operator(428ea8e3-b433-48c4-9330-9f4ace89b6fa): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Sep 09 20:50:23.241316 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:23.241269 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"iptables-alerter\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-network-operator/iptables-alerter-ppc2l" podUID="428ea8e3-b433-48c4-9330-9f4ace89b6fa" Sep 09 20:50:23.500685 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:23.500616 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91f58455-46ed-4617-bfe6-ef79a8fc05f3-original-pull-secret\") pod \"global-pull-secret-syncer-csblg\" (UID: \"91f58455-46ed-4617-bfe6-ef79a8fc05f3\") " pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:23.500807 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:23.500700 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs\") pod \"network-metrics-daemon-9dfd9\" (UID: \"f8e8b1d2-6523-403e-bace-b96b38d8449b\") " pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:23.500807 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:23.500763 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 20:50:23.500807 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:23.500794 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:50:23.500954 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:23.500832 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91f58455-46ed-4617-bfe6-ef79a8fc05f3-original-pull-secret podName:91f58455-46ed-4617-bfe6-ef79a8fc05f3 nodeName:}" failed. No retries permitted until 2026-09-09 20:50:24.500813363 +0000 UTC m=+3.107056067 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91f58455-46ed-4617-bfe6-ef79a8fc05f3-original-pull-secret") pod "global-pull-secret-syncer-csblg" (UID: "91f58455-46ed-4617-bfe6-ef79a8fc05f3") : object "kube-system"/"original-pull-secret" not registered Sep 09 20:50:23.500954 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:23.500850 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs podName:f8e8b1d2-6523-403e-bace-b96b38d8449b nodeName:}" failed. No retries permitted until 2026-09-09 20:50:24.500840881 +0000 UTC m=+3.107083574 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs") pod "network-metrics-daemon-9dfd9" (UID: "f8e8b1d2-6523-403e-bace-b96b38d8449b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:50:24.747573 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:24.747546 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzjcn\" (UniqueName: \"kubernetes.io/projected/b5e2d5ce-182b-46ef-b30d-52e8115c4f54-kube-api-access-bzjcn\") pod \"network-check-target-tbqnj\" (UID: \"b5e2d5ce-182b-46ef-b30d-52e8115c4f54\") " pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:24.747740 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:24.747719 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 20:50:24.747807 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:24.747749 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 20:50:24.747807 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:24.747762 2580 projected.go:196] Error preparing data for projected volume kube-api-access-bzjcn for pod openshift-network-diagnostics/network-check-target-tbqnj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:50:24.747926 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:24.747815 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5e2d5ce-182b-46ef-b30d-52e8115c4f54-kube-api-access-bzjcn podName:b5e2d5ce-182b-46ef-b30d-52e8115c4f54 nodeName:}" failed. No retries permitted until 2026-09-09 20:50:25.74779781 +0000 UTC m=+3.208439712 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bzjcn" (UniqueName: "kubernetes.io/projected/b5e2d5ce-182b-46ef-b30d-52e8115c4f54-kube-api-access-bzjcn") pod "network-check-target-tbqnj" (UID: "b5e2d5ce-182b-46ef-b30d-52e8115c4f54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:50:24.820127 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:24.820101 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 20:50:24.982839 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:24.982321 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:45:23 +0000 UTC" deadline="2028-06-17 17:46:40.302509704 +0000 UTC" Sep 09 20:50:24.982839 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:24.982376 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15524h56m15.320138392s" Sep 09 20:50:25.038787 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:25.038765 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:25.038960 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:25.038884 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csblg" podUID="91f58455-46ed-4617-bfe6-ef79a8fc05f3" Sep 09 20:50:25.061906 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:25.061851 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-95fw4" event={"ID":"b8cbb806-0e1f-40f8-9595-3001333438ea","Type":"ContainerStarted","Data":"e5f71b9bdf43e3f4da75ab8e5b69404de677f16683e19ed0a0255ed0bbe1db0b"} Sep 09 20:50:25.068571 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:25.068539 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" event={"ID":"9c859f8c-e68a-4579-966e-8e01d3f66928","Type":"ContainerStarted","Data":"e62a0a74e798b45d443d504e090b9e1f40ae9bdd098f19e4bd9f5d656bf39791"} Sep 09 20:50:25.090648 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:25.090596 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" event={"ID":"ddb3d74a-a1bb-4bda-8d0d-89320e50f357","Type":"ContainerStarted","Data":"cf3514026a11bdf7d07ca7da3b5d91647fbce5f62ada60e412046f1709f051a3"} Sep 09 20:50:25.115846 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:25.115813 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gx8lm" event={"ID":"e64d1b19-5ae3-4758-8753-e83a736855f0","Type":"ContainerStarted","Data":"878718beecc9be0912cf6febe13c53a867c9069d8c756613ca7b3fd74f36b7f7"} Sep 09 20:50:25.125632 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:25.125605 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bxfjt" event={"ID":"717799e6-8e3b-4a16-88d6-384d950d5385","Type":"ContainerStarted","Data":"0434e985a6a45e98375c01773e358fd2ea0e06ac2eeb3f232d279ad2be451d70"} Sep 09 20:50:25.131945 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:25.131918 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" event={"ID":"c9d9f144-f418-422d-9356-b682af358e46","Type":"ContainerStarted","Data":"c0643926ad444cbae7a3077f2c8193ce4e2efe7a1906a0bc54e8a014058e4a84"} Sep 09 20:50:25.133784 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:25.133756 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2g5nr" event={"ID":"f5167c4c-e0f9-496f-b62f-78764ba3cbd7","Type":"ContainerStarted","Data":"217fb274b4178cfdcad3bf08b2498ba567f7e2fd5a47555d0606c4fd00ef98b4"} Sep 09 20:50:25.136730 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:25.136681 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal" event={"ID":"99981a6272193d33eff6882647ea56fb","Type":"ContainerStarted","Data":"96d4501bbcb30b7ab4a8580872d3ab4fa49f52d81dfbd93bf836c4dc5b74c537"} Sep 09 20:50:25.139215 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:25.139190 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ppc2l" event={"ID":"428ea8e3-b433-48c4-9330-9f4ace89b6fa","Type":"ContainerStarted","Data":"53150155988d899a061614948c83a48f5794c1d81b6a0cc6ed9accc2c0602e6e"} Sep 09 20:50:25.142152 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:25.142123 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"iptables-alerter\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:40bc1b9c6b608369978e9f165c5e22d7a75fd91b3480cba60cd275228d721616\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-network-operator/iptables-alerter-ppc2l" podUID="428ea8e3-b433-48c4-9330-9f4ace89b6fa" Sep 09 20:50:25.142854 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:25.142809 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rwtrs" event={"ID":"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85","Type":"ContainerStarted","Data":"147f9ccc144b42be2325abb4113e5447ac93f185f2020ab55c95072753eec499"} Sep 09 20:50:25.146590 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:25.146569 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-19.ec2.internal" event={"ID":"70b70169d86c3bcf2d0a1e114ed46d9a","Type":"ContainerStarted","Data":"aab547ce524fe1606845216935c44d4477cf262d37b779122bb2c34f38b1bc6c"} Sep 09 20:50:25.654284 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:25.654250 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs\") pod \"network-metrics-daemon-9dfd9\" (UID: \"f8e8b1d2-6523-403e-bace-b96b38d8449b\") " pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:25.654492 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:25.654308 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91f58455-46ed-4617-bfe6-ef79a8fc05f3-original-pull-secret\") pod \"global-pull-secret-syncer-csblg\" (UID: \"91f58455-46ed-4617-bfe6-ef79a8fc05f3\") " pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:25.654492 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:25.654409 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:50:25.654492 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:25.654445 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 20:50:25.654492 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:25.654476 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs podName:f8e8b1d2-6523-403e-bace-b96b38d8449b nodeName:}" failed. No retries permitted until 2026-09-09 20:50:27.654456346 +0000 UTC m=+5.115098247 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs") pod "network-metrics-daemon-9dfd9" (UID: "f8e8b1d2-6523-403e-bace-b96b38d8449b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:50:25.654691 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:25.654498 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91f58455-46ed-4617-bfe6-ef79a8fc05f3-original-pull-secret podName:91f58455-46ed-4617-bfe6-ef79a8fc05f3 nodeName:}" failed. No retries permitted until 2026-09-09 20:50:27.654483415 +0000 UTC m=+5.115125326 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91f58455-46ed-4617-bfe6-ef79a8fc05f3-original-pull-secret") pod "global-pull-secret-syncer-csblg" (UID: "91f58455-46ed-4617-bfe6-ef79a8fc05f3") : object "kube-system"/"original-pull-secret" not registered Sep 09 20:50:25.754929 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:25.754882 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzjcn\" (UniqueName: \"kubernetes.io/projected/b5e2d5ce-182b-46ef-b30d-52e8115c4f54-kube-api-access-bzjcn\") pod \"network-check-target-tbqnj\" (UID: \"b5e2d5ce-182b-46ef-b30d-52e8115c4f54\") " pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:25.755112 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:25.755084 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 20:50:25.755171 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:25.755148 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 20:50:25.755171 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:25.755163 2580 projected.go:196] Error preparing data for projected volume kube-api-access-bzjcn for pod openshift-network-diagnostics/network-check-target-tbqnj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:50:25.755270 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:25.755218 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5e2d5ce-182b-46ef-b30d-52e8115c4f54-kube-api-access-bzjcn podName:b5e2d5ce-182b-46ef-b30d-52e8115c4f54 nodeName:}" failed. No retries permitted until 2026-09-09 20:50:27.755200731 +0000 UTC m=+5.215842646 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bzjcn" (UniqueName: "kubernetes.io/projected/b5e2d5ce-182b-46ef-b30d-52e8115c4f54-kube-api-access-bzjcn") pod "network-check-target-tbqnj" (UID: "b5e2d5ce-182b-46ef-b30d-52e8115c4f54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:50:25.778055 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:25.778028 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 20:50:25.982791 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:25.982732 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:45:23 +0000 UTC" deadline="2028-06-25 01:10:21.310907894 +0000 UTC" Sep 09 20:50:25.982791 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:25.982789 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15700h19m55.328122462s" Sep 09 20:50:26.038081 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:26.038056 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:26.038214 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:26.038186 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dfd9" podUID="f8e8b1d2-6523-403e-bace-b96b38d8449b" Sep 09 20:50:26.038287 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:26.038260 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:26.038338 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:26.038328 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbqnj" podUID="b5e2d5ce-182b-46ef-b30d-52e8115c4f54" Sep 09 20:50:27.038557 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:27.038069 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:27.038557 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:27.038188 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csblg" podUID="91f58455-46ed-4617-bfe6-ef79a8fc05f3" Sep 09 20:50:27.670162 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:27.670126 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs\") pod \"network-metrics-daemon-9dfd9\" (UID: \"f8e8b1d2-6523-403e-bace-b96b38d8449b\") " pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:27.670358 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:27.670173 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91f58455-46ed-4617-bfe6-ef79a8fc05f3-original-pull-secret\") pod \"global-pull-secret-syncer-csblg\" (UID: \"91f58455-46ed-4617-bfe6-ef79a8fc05f3\") " pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:27.670358 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:27.670301 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 20:50:27.670552 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:27.670381 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91f58455-46ed-4617-bfe6-ef79a8fc05f3-original-pull-secret podName:91f58455-46ed-4617-bfe6-ef79a8fc05f3 nodeName:}" failed. No retries permitted until 2026-09-09 20:50:31.670363422 +0000 UTC m=+9.131005322 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91f58455-46ed-4617-bfe6-ef79a8fc05f3-original-pull-secret") pod "global-pull-secret-syncer-csblg" (UID: "91f58455-46ed-4617-bfe6-ef79a8fc05f3") : object "kube-system"/"original-pull-secret" not registered Sep 09 20:50:27.670826 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:27.670792 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:50:27.670913 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:27.670857 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs podName:f8e8b1d2-6523-403e-bace-b96b38d8449b nodeName:}" failed. No retries permitted until 2026-09-09 20:50:31.670846617 +0000 UTC m=+9.131488518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs") pod "network-metrics-daemon-9dfd9" (UID: "f8e8b1d2-6523-403e-bace-b96b38d8449b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:50:27.770929 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:27.770800 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzjcn\" (UniqueName: \"kubernetes.io/projected/b5e2d5ce-182b-46ef-b30d-52e8115c4f54-kube-api-access-bzjcn\") pod \"network-check-target-tbqnj\" (UID: \"b5e2d5ce-182b-46ef-b30d-52e8115c4f54\") " pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:27.771101 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:27.770979 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 20:50:27.771101 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:27.770995 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 20:50:27.771101 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:27.771003 2580 projected.go:196] Error preparing data for projected volume kube-api-access-bzjcn for pod openshift-network-diagnostics/network-check-target-tbqnj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:50:27.771101 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:27.771039 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5e2d5ce-182b-46ef-b30d-52e8115c4f54-kube-api-access-bzjcn podName:b5e2d5ce-182b-46ef-b30d-52e8115c4f54 nodeName:}" failed. No retries permitted until 2026-09-09 20:50:31.771028315 +0000 UTC m=+9.231670218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bzjcn" (UniqueName: "kubernetes.io/projected/b5e2d5ce-182b-46ef-b30d-52e8115c4f54-kube-api-access-bzjcn") pod "network-check-target-tbqnj" (UID: "b5e2d5ce-182b-46ef-b30d-52e8115c4f54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:50:28.037403 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:28.037371 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:28.037571 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:28.037511 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dfd9" podUID="f8e8b1d2-6523-403e-bace-b96b38d8449b" Sep 09 20:50:28.037915 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:28.037371 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:28.038081 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:28.038051 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbqnj" podUID="b5e2d5ce-182b-46ef-b30d-52e8115c4f54" Sep 09 20:50:29.039195 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:29.038672 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:29.039195 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:29.038799 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csblg" podUID="91f58455-46ed-4617-bfe6-ef79a8fc05f3" Sep 09 20:50:30.038835 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:30.038231 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:30.038835 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:30.038345 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbqnj" podUID="b5e2d5ce-182b-46ef-b30d-52e8115c4f54" Sep 09 20:50:30.038835 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:30.038697 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:30.038835 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:30.038791 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dfd9" podUID="f8e8b1d2-6523-403e-bace-b96b38d8449b" Sep 09 20:50:31.039873 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:31.039673 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:31.040239 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:31.039973 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csblg" podUID="91f58455-46ed-4617-bfe6-ef79a8fc05f3" Sep 09 20:50:31.702815 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:31.702782 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs\") pod \"network-metrics-daemon-9dfd9\" (UID: \"f8e8b1d2-6523-403e-bace-b96b38d8449b\") " pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:31.703030 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:31.702839 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91f58455-46ed-4617-bfe6-ef79a8fc05f3-original-pull-secret\") pod \"global-pull-secret-syncer-csblg\" (UID: \"91f58455-46ed-4617-bfe6-ef79a8fc05f3\") " pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:31.703030 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:31.702918 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:50:31.703030 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:31.702989 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs podName:f8e8b1d2-6523-403e-bace-b96b38d8449b nodeName:}" failed. No retries permitted until 2026-09-09 20:50:39.702973923 +0000 UTC m=+17.163615827 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs") pod "network-metrics-daemon-9dfd9" (UID: "f8e8b1d2-6523-403e-bace-b96b38d8449b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:50:31.703030 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:31.703011 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 20:50:31.703306 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:31.703065 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91f58455-46ed-4617-bfe6-ef79a8fc05f3-original-pull-secret podName:91f58455-46ed-4617-bfe6-ef79a8fc05f3 nodeName:}" failed. No retries permitted until 2026-09-09 20:50:39.703049787 +0000 UTC m=+17.163691699 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91f58455-46ed-4617-bfe6-ef79a8fc05f3-original-pull-secret") pod "global-pull-secret-syncer-csblg" (UID: "91f58455-46ed-4617-bfe6-ef79a8fc05f3") : object "kube-system"/"original-pull-secret" not registered Sep 09 20:50:31.804239 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:31.804171 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzjcn\" (UniqueName: \"kubernetes.io/projected/b5e2d5ce-182b-46ef-b30d-52e8115c4f54-kube-api-access-bzjcn\") pod \"network-check-target-tbqnj\" (UID: \"b5e2d5ce-182b-46ef-b30d-52e8115c4f54\") " pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:31.804385 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:31.804369 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 20:50:31.804481 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:31.804402 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 20:50:31.804481 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:31.804416 2580 projected.go:196] Error preparing data for projected volume kube-api-access-bzjcn for pod openshift-network-diagnostics/network-check-target-tbqnj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:50:31.804577 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:31.804481 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5e2d5ce-182b-46ef-b30d-52e8115c4f54-kube-api-access-bzjcn podName:b5e2d5ce-182b-46ef-b30d-52e8115c4f54 nodeName:}" failed. No retries permitted until 2026-09-09 20:50:39.804461933 +0000 UTC m=+17.265103847 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bzjcn" (UniqueName: "kubernetes.io/projected/b5e2d5ce-182b-46ef-b30d-52e8115c4f54-kube-api-access-bzjcn") pod "network-check-target-tbqnj" (UID: "b5e2d5ce-182b-46ef-b30d-52e8115c4f54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:50:32.038392 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:32.038351 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:32.038607 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:32.038493 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbqnj" podUID="b5e2d5ce-182b-46ef-b30d-52e8115c4f54" Sep 09 20:50:32.038948 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:32.038846 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:32.039058 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:32.038983 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dfd9" podUID="f8e8b1d2-6523-403e-bace-b96b38d8449b" Sep 09 20:50:33.039257 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:33.038867 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:33.039257 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:33.038971 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csblg" podUID="91f58455-46ed-4617-bfe6-ef79a8fc05f3" Sep 09 20:50:34.038012 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:34.037982 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:34.038192 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:34.038021 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:34.038192 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:34.038086 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbqnj" podUID="b5e2d5ce-182b-46ef-b30d-52e8115c4f54" Sep 09 20:50:34.038298 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:34.038217 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dfd9" podUID="f8e8b1d2-6523-403e-bace-b96b38d8449b" Sep 09 20:50:35.037807 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:35.037778 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:35.038167 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:35.037915 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csblg" podUID="91f58455-46ed-4617-bfe6-ef79a8fc05f3" Sep 09 20:50:36.037832 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:36.037799 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:36.038260 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:36.037837 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:36.038260 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:36.037935 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dfd9" podUID="f8e8b1d2-6523-403e-bace-b96b38d8449b" Sep 09 20:50:36.038260 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:36.038048 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbqnj" podUID="b5e2d5ce-182b-46ef-b30d-52e8115c4f54" Sep 09 20:50:37.038058 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:37.038025 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:37.038434 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:37.038137 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csblg" podUID="91f58455-46ed-4617-bfe6-ef79a8fc05f3" Sep 09 20:50:38.037615 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:38.037585 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:38.037805 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:38.037709 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dfd9" podUID="f8e8b1d2-6523-403e-bace-b96b38d8449b" Sep 09 20:50:38.037805 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:38.037765 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:38.037949 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:38.037847 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbqnj" podUID="b5e2d5ce-182b-46ef-b30d-52e8115c4f54" Sep 09 20:50:39.038363 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:39.038331 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:39.038749 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:39.038450 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csblg" podUID="91f58455-46ed-4617-bfe6-ef79a8fc05f3" Sep 09 20:50:39.766604 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:39.766573 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs\") pod \"network-metrics-daemon-9dfd9\" (UID: \"f8e8b1d2-6523-403e-bace-b96b38d8449b\") " pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:39.766864 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:39.766618 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91f58455-46ed-4617-bfe6-ef79a8fc05f3-original-pull-secret\") pod \"global-pull-secret-syncer-csblg\" (UID: \"91f58455-46ed-4617-bfe6-ef79a8fc05f3\") " pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:39.766864 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:39.766725 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 20:50:39.766864 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:39.766728 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:50:39.766864 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:39.766782 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91f58455-46ed-4617-bfe6-ef79a8fc05f3-original-pull-secret podName:91f58455-46ed-4617-bfe6-ef79a8fc05f3 nodeName:}" failed. No retries permitted until 2026-09-09 20:50:55.766767551 +0000 UTC m=+33.227409452 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91f58455-46ed-4617-bfe6-ef79a8fc05f3-original-pull-secret") pod "global-pull-secret-syncer-csblg" (UID: "91f58455-46ed-4617-bfe6-ef79a8fc05f3") : object "kube-system"/"original-pull-secret" not registered Sep 09 20:50:39.766864 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:39.766796 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs podName:f8e8b1d2-6523-403e-bace-b96b38d8449b nodeName:}" failed. No retries permitted until 2026-09-09 20:50:55.766789638 +0000 UTC m=+33.227431539 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs") pod "network-metrics-daemon-9dfd9" (UID: "f8e8b1d2-6523-403e-bace-b96b38d8449b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:50:39.867036 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:39.867009 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzjcn\" (UniqueName: \"kubernetes.io/projected/b5e2d5ce-182b-46ef-b30d-52e8115c4f54-kube-api-access-bzjcn\") pod \"network-check-target-tbqnj\" (UID: \"b5e2d5ce-182b-46ef-b30d-52e8115c4f54\") " pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:39.867211 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:39.867185 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 20:50:39.867211 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:39.867205 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 20:50:39.867315 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:39.867216 2580 projected.go:196] Error preparing data for projected volume kube-api-access-bzjcn for pod openshift-network-diagnostics/network-check-target-tbqnj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:50:39.867315 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:39.867260 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5e2d5ce-182b-46ef-b30d-52e8115c4f54-kube-api-access-bzjcn podName:b5e2d5ce-182b-46ef-b30d-52e8115c4f54 nodeName:}" failed. No retries permitted until 2026-09-09 20:50:55.867243523 +0000 UTC m=+33.327885423 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bzjcn" (UniqueName: "kubernetes.io/projected/b5e2d5ce-182b-46ef-b30d-52e8115c4f54-kube-api-access-bzjcn") pod "network-check-target-tbqnj" (UID: "b5e2d5ce-182b-46ef-b30d-52e8115c4f54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:50:40.037784 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:40.037708 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:40.037784 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:40.037735 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:40.038009 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:40.037934 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dfd9" podUID="f8e8b1d2-6523-403e-bace-b96b38d8449b" Sep 09 20:50:40.038065 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:40.038046 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbqnj" podUID="b5e2d5ce-182b-46ef-b30d-52e8115c4f54" Sep 09 20:50:41.038298 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:41.038249 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:41.038745 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:41.038361 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csblg" podUID="91f58455-46ed-4617-bfe6-ef79a8fc05f3" Sep 09 20:50:42.038162 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:42.038131 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:42.038342 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:42.038254 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dfd9" podUID="f8e8b1d2-6523-403e-bace-b96b38d8449b" Sep 09 20:50:42.038342 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:42.038132 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:42.038342 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:42.038335 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbqnj" podUID="b5e2d5ce-182b-46ef-b30d-52e8115c4f54" Sep 09 20:50:43.042267 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:43.040258 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:43.042267 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:43.040560 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csblg" podUID="91f58455-46ed-4617-bfe6-ef79a8fc05f3" Sep 09 20:50:43.190046 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:43.189854 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-19.ec2.internal" event={"ID":"70b70169d86c3bcf2d0a1e114ed46d9a","Type":"ContainerStarted","Data":"769110970a14fe8cef7723929998ddd8146d38cdf2b9c1733da2d69d6eb2a84a"} Sep 09 20:50:43.196973 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:43.196943 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-95fw4" event={"ID":"b8cbb806-0e1f-40f8-9595-3001333438ea","Type":"ContainerStarted","Data":"ae54e3b6792a4a3715ef81b010347603e6678545057b94a3b78c35df321c676d"} Sep 09 20:50:43.203418 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:43.203211 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-19.ec2.internal" podStartSLOduration=20.203192769 podStartE2EDuration="20.203192769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:43.202490706 +0000 UTC m=+20.663132629" watchObservedRunningTime="2026-09-09 20:50:43.203192769 +0000 UTC m=+20.663834692" Sep 09 20:50:43.204091 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:43.204023 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" event={"ID":"9c859f8c-e68a-4579-966e-8e01d3f66928","Type":"ContainerStarted","Data":"82c26f6ad1d917e8fc6f00622bba4b86ae98ea0fce6da50a00ebf8f14ad7646c"} Sep 09 20:50:43.218082 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:43.218021 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-95fw4" podStartSLOduration=1.672065784 podStartE2EDuration="20.218001319s" totalImagesPullingTime="18.545935535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:43.217155912 +0000 UTC m=+20.677797836" watchObservedRunningTime="2026-09-09 20:50:43.218001319 +0000 UTC m=+20.678643238" Sep 09 20:50:43.222537 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:43.222518 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tctqv_ddb3d74a-a1bb-4bda-8d0d-89320e50f357/ovn-acl-logging/0.log" Sep 09 20:50:43.227199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:43.226873 2580 generic.go:356] "Generic (PLEG): container finished" podID="ddb3d74a-a1bb-4bda-8d0d-89320e50f357" containerID="4bee547c729108ddd02e537684fa0deff230193172c6148d92646818052f9e85" exitCode=1 Sep 09 20:50:43.227199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:43.226947 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" event={"ID":"ddb3d74a-a1bb-4bda-8d0d-89320e50f357","Type":"ContainerStarted","Data":"53bd853280a93f394a800c7aefb54a863b401fa5162048ce9225c21cc2199a6d"} Sep 09 20:50:43.227199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:43.227018 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" event={"ID":"ddb3d74a-a1bb-4bda-8d0d-89320e50f357","Type":"ContainerStarted","Data":"05c5e3c0b7d5226650a6eed6a409625c7290bc84498f8a5d78f4a5acef302e6b"} Sep 09 20:50:43.227199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:43.227067 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" event={"ID":"ddb3d74a-a1bb-4bda-8d0d-89320e50f357","Type":"ContainerDied","Data":"4bee547c729108ddd02e537684fa0deff230193172c6148d92646818052f9e85"} Sep 09 20:50:43.227199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:43.227118 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" event={"ID":"ddb3d74a-a1bb-4bda-8d0d-89320e50f357","Type":"ContainerStarted","Data":"c6b831a885bed6d27cb9858c74a157cd8b402b2cfa57df8358832c061a8e90de"} Sep 09 20:50:43.234807 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:43.234366 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-c5dnk" podStartSLOduration=2.067032597 podStartE2EDuration="20.234350042s" totalImagesPullingTime="18.167317445s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:43.233881442 +0000 UTC m=+20.694523348" watchObservedRunningTime="2026-09-09 20:50:43.234350042 +0000 UTC m=+20.694991965" Sep 09 20:50:44.038218 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:44.038004 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:44.038348 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:44.038075 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:44.038348 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:44.038314 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbqnj" podUID="b5e2d5ce-182b-46ef-b30d-52e8115c4f54" Sep 09 20:50:44.038457 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:44.038401 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dfd9" podUID="f8e8b1d2-6523-403e-bace-b96b38d8449b" Sep 09 20:50:44.230636 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:44.230605 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ppc2l" event={"ID":"428ea8e3-b433-48c4-9330-9f4ace89b6fa","Type":"ContainerStarted","Data":"9f4fdb60f7292af3b74d2f1d5fbacde4787672171f9f8153904e6d16d3ab328c"} Sep 09 20:50:44.232183 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:44.232159 2580 generic.go:356] "Generic (PLEG): container finished" podID="75366d82-a9b6-4a08-a9cf-a3c9cd38ef85" containerID="45ff5e7cb7615df91fbd01e1804ac7708d1ade42c5ab2681d0436edfd32b2379" exitCode=0 Sep 09 20:50:44.232308 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:44.232249 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rwtrs" event={"ID":"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85","Type":"ContainerDied","Data":"45ff5e7cb7615df91fbd01e1804ac7708d1ade42c5ab2681d0436edfd32b2379"} Sep 09 20:50:44.238732 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:44.238715 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tctqv_ddb3d74a-a1bb-4bda-8d0d-89320e50f357/ovn-acl-logging/0.log" Sep 09 20:50:44.239164 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:44.239141 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" event={"ID":"ddb3d74a-a1bb-4bda-8d0d-89320e50f357","Type":"ContainerStarted","Data":"d5a0f060195dddc6e4ddbcd2238b4a5aef752ffdb29801b8f597ed8931c4e947"} Sep 09 20:50:44.239298 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:44.239280 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" event={"ID":"ddb3d74a-a1bb-4bda-8d0d-89320e50f357","Type":"ContainerStarted","Data":"90f67142808a9643c92a5e6ab35a85cdd86d431c03986c1ee7bd5fc5bd884fac"} Sep 09 20:50:44.240720 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:44.240699 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gx8lm" event={"ID":"e64d1b19-5ae3-4758-8753-e83a736855f0","Type":"ContainerStarted","Data":"d92125e40367ee15bc926b3f1e08f1d94d697f734ad93fedea6bfa7dea22d486"} Sep 09 20:50:44.242202 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:44.242180 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bxfjt" event={"ID":"717799e6-8e3b-4a16-88d6-384d950d5385","Type":"ContainerStarted","Data":"b283a9d461765ac507c4c5157c4e9f58fa78ba762ead22cd03d8f918768bae4e"} Sep 09 20:50:44.243714 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:44.243696 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" event={"ID":"c9d9f144-f418-422d-9356-b682af358e46","Type":"ContainerStarted","Data":"46cb72f757e132ddcaf786095de2bbf4db8409fd96db96558e847f6e5b4ffdec"} Sep 09 20:50:44.244756 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:44.244716 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ppc2l" podStartSLOduration=21.24470133 podStartE2EDuration="21.24470133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-09 20:50:44.243977934 +0000 UTC m=+21.704619894" watchObservedRunningTime="2026-09-09 20:50:44.24470133 +0000 UTC m=+21.705343252" Sep 09 20:50:44.245283 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:44.245261 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2g5nr" event={"ID":"f5167c4c-e0f9-496f-b62f-78764ba3cbd7","Type":"ContainerStarted","Data":"e1e0b237c12a456b5a7ae33b7fabc362836074e1a937371eac54037982b8e6d6"} Sep 09 20:50:44.247074 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:44.247052 2580 generic.go:356] "Generic (PLEG): container finished" podID="99981a6272193d33eff6882647ea56fb" containerID="373bc89624df4c6e5cb8dc648838bf66d9f2d9eabe7ed18b7ee8b7fff543d910" exitCode=0 Sep 09 20:50:44.247166 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:44.247139 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal" event={"ID":"99981a6272193d33eff6882647ea56fb","Type":"ContainerDied","Data":"373bc89624df4c6e5cb8dc648838bf66d9f2d9eabe7ed18b7ee8b7fff543d910"} Sep 09 20:50:44.261002 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:44.260947 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gx8lm" podStartSLOduration=3.088632813 podStartE2EDuration="21.260932402s" totalImagesPullingTime="18.172299589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:44.260818287 +0000 UTC m=+21.721460209" watchObservedRunningTime="2026-09-09 20:50:44.260932402 +0000 UTC m=+21.721574323" Sep 09 20:50:44.292773 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:44.292737 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bxfjt" podStartSLOduration=3.068065004 podStartE2EDuration="21.292725772s" totalImagesPullingTime="18.224660768s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:44.292654719 +0000 UTC m=+21.753296640" watchObservedRunningTime="2026-09-09 20:50:44.292725772 +0000 UTC m=+21.753367693" Sep 09 20:50:44.334670 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:44.334630 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2g5nr" podStartSLOduration=3.070999942 podStartE2EDuration="21.334617713s" totalImagesPullingTime="18.263617771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:44.318386467 +0000 UTC m=+21.779028390" watchObservedRunningTime="2026-09-09 20:50:44.334617713 +0000 UTC m=+21.795259635" Sep 09 20:50:44.505516 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:44.505490 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 09 20:50:45.009762 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:45.009669 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T20:50:44.505504563Z","UUID":"3b2e51d4-21d5-47a7-bf0c-1c51837bb007","Handler":null,"Name":"","Endpoint":""} Sep 09 20:50:45.011434 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:45.011401 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 09 20:50:45.011434 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:45.011436 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 09 20:50:45.037820 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:45.037793 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:45.037978 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:45.037929 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csblg" podUID="91f58455-46ed-4617-bfe6-ef79a8fc05f3" Sep 09 20:50:45.251631 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:45.251545 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" event={"ID":"c9d9f144-f418-422d-9356-b682af358e46","Type":"ContainerStarted","Data":"3873f2aecefc107567fbd771b5ef466d616694e578e70e7da001c6c9c87ee662"} Sep 09 20:50:45.253825 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:45.253793 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal" event={"ID":"99981a6272193d33eff6882647ea56fb","Type":"ContainerStarted","Data":"e5a5b61ab3aa227fbe0e58c849cc708b323d9ccb5644b762416c9382eb518332"} Sep 09 20:50:45.270732 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:45.270652 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal" podStartSLOduration=22.270639403 podStartE2EDuration="22.270639403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:45.270273597 +0000 UTC m=+22.730915520" watchObservedRunningTime="2026-09-09 20:50:45.270639403 +0000 UTC m=+22.731281325" Sep 09 20:50:46.038417 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:46.038384 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:46.038588 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:46.038490 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbqnj" podUID="b5e2d5ce-182b-46ef-b30d-52e8115c4f54" Sep 09 20:50:46.038588 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:46.038540 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:46.038681 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:46.038632 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dfd9" podUID="f8e8b1d2-6523-403e-bace-b96b38d8449b" Sep 09 20:50:46.259179 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:46.259003 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" event={"ID":"c9d9f144-f418-422d-9356-b682af358e46","Type":"ContainerStarted","Data":"05f3a01ebb0b8ccb9d64366f964ccb3719b2125bdd3a4b071e39d79cf8bbcf38"} Sep 09 20:50:46.262612 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:46.262592 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tctqv_ddb3d74a-a1bb-4bda-8d0d-89320e50f357/ovn-acl-logging/0.log" Sep 09 20:50:46.263104 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:46.263079 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" event={"ID":"ddb3d74a-a1bb-4bda-8d0d-89320e50f357","Type":"ContainerStarted","Data":"233e713f76e8b425e63d1d7bb52f25082968255e2567c2eb39fcf05dd66411bc"} Sep 09 20:50:46.275403 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:46.275361 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fqlj" podStartSLOduration=3.0913203 podStartE2EDuration="23.275343034s" totalImagesPullingTime="20.184022734s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:50:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:46.275292523 +0000 UTC m=+23.735934446" watchObservedRunningTime="2026-09-09 20:50:46.275343034 +0000 UTC m=+23.735984945" Sep 09 20:50:46.530113 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:46.530086 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2g5nr" Sep 09 20:50:46.530677 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:46.530656 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2g5nr" Sep 09 20:50:47.037497 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:47.037465 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:47.037685 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:47.037588 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csblg" podUID="91f58455-46ed-4617-bfe6-ef79a8fc05f3" Sep 09 20:50:47.265356 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:47.265311 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2g5nr" Sep 09 20:50:47.265788 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:47.265673 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2g5nr" Sep 09 20:50:48.037823 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:48.037796 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:48.038026 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:48.037942 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dfd9" podUID="f8e8b1d2-6523-403e-bace-b96b38d8449b" Sep 09 20:50:48.038026 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:48.038001 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:48.038142 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:48.038093 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbqnj" podUID="b5e2d5ce-182b-46ef-b30d-52e8115c4f54" Sep 09 20:50:49.038743 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:49.038501 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:49.039038 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:49.038808 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csblg" podUID="91f58455-46ed-4617-bfe6-ef79a8fc05f3" Sep 09 20:50:49.270477 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:49.270453 2580 generic.go:356] "Generic (PLEG): container finished" podID="75366d82-a9b6-4a08-a9cf-a3c9cd38ef85" containerID="3dd2b99c7d502a05cf1fbe6bc0befeb34ab03fa97cffdb2daa84386279f0bbc7" exitCode=0 Sep 09 20:50:49.270558 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:49.270525 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rwtrs" event={"ID":"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85","Type":"ContainerDied","Data":"3dd2b99c7d502a05cf1fbe6bc0befeb34ab03fa97cffdb2daa84386279f0bbc7"} Sep 09 20:50:49.274178 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:49.274162 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tctqv_ddb3d74a-a1bb-4bda-8d0d-89320e50f357/ovn-acl-logging/0.log" Sep 09 20:50:49.274560 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:49.274540 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" event={"ID":"ddb3d74a-a1bb-4bda-8d0d-89320e50f357","Type":"ContainerStarted","Data":"90b6f8b2e95da5012e443bd833dd1b8365ba4986576af30f646b2d960bbb035c"} Sep 09 20:50:49.274719 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:49.274705 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:49.274778 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:49.274746 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:49.274934 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:49.274918 2580 scope.go:117] "RemoveContainer" containerID="4bee547c729108ddd02e537684fa0deff230193172c6148d92646818052f9e85" Sep 09 20:50:49.290377 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:49.290284 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:50.037747 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:50.037710 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:50.037747 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:50.037739 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:50.037940 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:50.037841 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dfd9" podUID="f8e8b1d2-6523-403e-bace-b96b38d8449b" Sep 09 20:50:50.037994 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:50.037958 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbqnj" podUID="b5e2d5ce-182b-46ef-b30d-52e8115c4f54" Sep 09 20:50:50.278959 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:50.278935 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rwtrs" event={"ID":"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85","Type":"ContainerStarted","Data":"949ada2306fa9577ccb5a02c26d5572af2ef2b47828909f6f69602f751965819"} Sep 09 20:50:50.282774 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:50.282756 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tctqv_ddb3d74a-a1bb-4bda-8d0d-89320e50f357/ovn-acl-logging/0.log" Sep 09 20:50:50.286175 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:50.286150 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" event={"ID":"ddb3d74a-a1bb-4bda-8d0d-89320e50f357","Type":"ContainerStarted","Data":"6000311069fff4237d20a7e0184c5bb6f8aa9cf700f02b8ad3ca87f6e2eec25a"} Sep 09 20:50:50.286528 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:50.286495 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:50.300144 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:50.300099 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:50:50.325444 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:50.325392 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" podStartSLOduration=9.107786776 podStartE2EDuration="27.325375765s" totalImagesPullingTime="18.217588989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:50.324839887 +0000 UTC m=+27.785481802" watchObservedRunningTime="2026-09-09 20:50:50.325375765 +0000 UTC m=+27.786017689" Sep 09 20:50:50.395031 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:50.395009 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-csblg"] Sep 09 20:50:50.395129 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:50.395091 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:50.395183 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:50.395167 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csblg" podUID="91f58455-46ed-4617-bfe6-ef79a8fc05f3" Sep 09 20:50:50.398257 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:50.398238 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tbqnj"] Sep 09 20:50:50.398352 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:50.398329 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:50.398462 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:50.398428 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbqnj" podUID="b5e2d5ce-182b-46ef-b30d-52e8115c4f54" Sep 09 20:50:50.398978 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:50.398958 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9dfd9"] Sep 09 20:50:50.399071 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:50.399058 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:50.399168 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:50.399148 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dfd9" podUID="f8e8b1d2-6523-403e-bace-b96b38d8449b" Sep 09 20:50:51.290498 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:51.290467 2580 generic.go:356] "Generic (PLEG): container finished" podID="75366d82-a9b6-4a08-a9cf-a3c9cd38ef85" containerID="949ada2306fa9577ccb5a02c26d5572af2ef2b47828909f6f69602f751965819" exitCode=0 Sep 09 20:50:51.290498 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:51.290495 2580 generic.go:356] "Generic (PLEG): container finished" podID="75366d82-a9b6-4a08-a9cf-a3c9cd38ef85" containerID="b2f51566882ac64fac59c318bbd2a79ebc1b560490239223427c896ef476f3ac" exitCode=0 Sep 09 20:50:51.290842 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:51.290543 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rwtrs" event={"ID":"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85","Type":"ContainerDied","Data":"949ada2306fa9577ccb5a02c26d5572af2ef2b47828909f6f69602f751965819"} Sep 09 20:50:51.290842 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:51.290588 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rwtrs" event={"ID":"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85","Type":"ContainerDied","Data":"b2f51566882ac64fac59c318bbd2a79ebc1b560490239223427c896ef476f3ac"} Sep 09 20:50:52.037412 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:52.037375 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:52.037572 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:52.037457 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:52.037572 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:52.037465 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbqnj" podUID="b5e2d5ce-182b-46ef-b30d-52e8115c4f54" Sep 09 20:50:52.037572 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:52.037548 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dfd9" podUID="f8e8b1d2-6523-403e-bace-b96b38d8449b" Sep 09 20:50:52.037572 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:52.037558 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:52.037744 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:52.037651 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csblg" podUID="91f58455-46ed-4617-bfe6-ef79a8fc05f3" Sep 09 20:50:54.038408 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:54.038228 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:54.038816 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:54.038250 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:54.038816 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:54.038259 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:54.038816 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:54.038526 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbqnj" podUID="b5e2d5ce-182b-46ef-b30d-52e8115c4f54" Sep 09 20:50:54.038816 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:54.038647 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-csblg" podUID="91f58455-46ed-4617-bfe6-ef79a8fc05f3" Sep 09 20:50:54.038816 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:54.038746 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dfd9" podUID="f8e8b1d2-6523-403e-bace-b96b38d8449b" Sep 09 20:50:55.423348 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:55.423318 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:50:55.519985 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:55.519959 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:50:55.790088 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:55.790068 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91f58455-46ed-4617-bfe6-ef79a8fc05f3-original-pull-secret\") pod \"global-pull-secret-syncer-csblg\" (UID: \"91f58455-46ed-4617-bfe6-ef79a8fc05f3\") " pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:55.790156 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:55.790117 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs\") pod \"network-metrics-daemon-9dfd9\" (UID: \"f8e8b1d2-6523-403e-bace-b96b38d8449b\") " pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:55.790221 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:55.790209 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:50:55.790261 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:55.790253 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs podName:f8e8b1d2-6523-403e-bace-b96b38d8449b nodeName:}" failed. No retries permitted until 2026-09-09 20:51:27.790241229 +0000 UTC m=+65.250883129 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs") pod "network-metrics-daemon-9dfd9" (UID: "f8e8b1d2-6523-403e-bace-b96b38d8449b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:50:55.790301 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:55.790209 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 20:50:55.790344 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:55.790325 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91f58455-46ed-4617-bfe6-ef79a8fc05f3-original-pull-secret podName:91f58455-46ed-4617-bfe6-ef79a8fc05f3 nodeName:}" failed. No retries permitted until 2026-09-09 20:51:27.790314255 +0000 UTC m=+65.250956158 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91f58455-46ed-4617-bfe6-ef79a8fc05f3-original-pull-secret") pod "global-pull-secret-syncer-csblg" (UID: "91f58455-46ed-4617-bfe6-ef79a8fc05f3") : object "kube-system"/"original-pull-secret" not registered Sep 09 20:50:55.878273 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:55.878252 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeReady" Sep 09 20:50:55.878374 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:55.878352 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 09 20:50:55.890760 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:55.890737 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzjcn\" (UniqueName: \"kubernetes.io/projected/b5e2d5ce-182b-46ef-b30d-52e8115c4f54-kube-api-access-bzjcn\") pod \"network-check-target-tbqnj\" (UID: \"b5e2d5ce-182b-46ef-b30d-52e8115c4f54\") " pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:55.890858 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:55.890847 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 20:50:55.890919 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:55.890872 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 20:50:55.890919 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:55.890881 2580 projected.go:196] Error preparing data for projected volume kube-api-access-bzjcn for pod openshift-network-diagnostics/network-check-target-tbqnj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:50:55.890986 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:55.890950 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5e2d5ce-182b-46ef-b30d-52e8115c4f54-kube-api-access-bzjcn podName:b5e2d5ce-182b-46ef-b30d-52e8115c4f54 nodeName:}" failed. No retries permitted until 2026-09-09 20:51:27.890939151 +0000 UTC m=+65.351581051 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bzjcn" (UniqueName: "kubernetes.io/projected/b5e2d5ce-182b-46ef-b30d-52e8115c4f54-kube-api-access-bzjcn") pod "network-check-target-tbqnj" (UID: "b5e2d5ce-182b-46ef-b30d-52e8115c4f54") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:50:55.932600 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:55.932566 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gb8k5"] Sep 09 20:50:55.958833 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:55.958806 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-77lm8"] Sep 09 20:50:55.958977 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:55.958965 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gb8k5" Sep 09 20:50:55.961312 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:55.961283 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 09 20:50:55.961422 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:55.961330 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wtvxz\"" Sep 09 20:50:55.961422 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:55.961340 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 09 20:50:55.972637 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:55.972619 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-77lm8"] Sep 09 20:50:55.972745 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:55.972651 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gb8k5"] Sep 09 20:50:55.972745 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:55.972730 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-77lm8" Sep 09 20:50:55.974707 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:55.974689 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 09 20:50:55.974830 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:55.974742 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-z287q\"" Sep 09 20:50:55.974830 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:55.974762 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 09 20:50:55.974961 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:55.974867 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 09 20:50:56.038342 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.038321 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:50:56.038441 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.038356 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:50:56.038441 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.038420 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:50:56.040951 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.040935 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 20:50:56.041052 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.040939 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kjjjq\"" Sep 09 20:50:56.041052 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.041034 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 20:50:56.041228 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.041215 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wskns\"" Sep 09 20:50:56.041290 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.041277 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 20:50:56.041393 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.041379 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 20:50:56.092083 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.092064 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-565l8\" (UniqueName: \"kubernetes.io/projected/bea0cf71-87de-42ff-bcef-683009062999-kube-api-access-565l8\") pod \"ingress-canary-77lm8\" (UID: \"bea0cf71-87de-42ff-bcef-683009062999\") " pod="openshift-ingress-canary/ingress-canary-77lm8" Sep 09 20:50:56.092168 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.092104 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81bd5151-ce78-4a7c-9ebf-b787d915430c-tmp-dir\") pod \"dns-default-gb8k5\" (UID: \"81bd5151-ce78-4a7c-9ebf-b787d915430c\") " pod="openshift-dns/dns-default-gb8k5" Sep 09 20:50:56.092168 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.092131 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls\") pod \"dns-default-gb8k5\" (UID: \"81bd5151-ce78-4a7c-9ebf-b787d915430c\") " pod="openshift-dns/dns-default-gb8k5" Sep 09 20:50:56.092168 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.092144 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bea0cf71-87de-42ff-bcef-683009062999-cert\") pod \"ingress-canary-77lm8\" (UID: \"bea0cf71-87de-42ff-bcef-683009062999\") " pod="openshift-ingress-canary/ingress-canary-77lm8" Sep 09 20:50:56.092321 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.092194 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/81bd5151-ce78-4a7c-9ebf-b787d915430c-config-volume\") pod \"dns-default-gb8k5\" (UID: \"81bd5151-ce78-4a7c-9ebf-b787d915430c\") " pod="openshift-dns/dns-default-gb8k5" Sep 09 20:50:56.092321 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.092269 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thcm2\" (UniqueName: \"kubernetes.io/projected/81bd5151-ce78-4a7c-9ebf-b787d915430c-kube-api-access-thcm2\") pod \"dns-default-gb8k5\" (UID: \"81bd5151-ce78-4a7c-9ebf-b787d915430c\") " pod="openshift-dns/dns-default-gb8k5" Sep 09 20:50:56.193290 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.193275 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-565l8\" (UniqueName: \"kubernetes.io/projected/bea0cf71-87de-42ff-bcef-683009062999-kube-api-access-565l8\") pod \"ingress-canary-77lm8\" (UID: \"bea0cf71-87de-42ff-bcef-683009062999\") " pod="openshift-ingress-canary/ingress-canary-77lm8" Sep 09 20:50:56.193371 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.193312 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81bd5151-ce78-4a7c-9ebf-b787d915430c-tmp-dir\") pod \"dns-default-gb8k5\" (UID: \"81bd5151-ce78-4a7c-9ebf-b787d915430c\") " pod="openshift-dns/dns-default-gb8k5" Sep 09 20:50:56.193371 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.193348 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls\") pod \"dns-default-gb8k5\" (UID: \"81bd5151-ce78-4a7c-9ebf-b787d915430c\") " pod="openshift-dns/dns-default-gb8k5" Sep 09 20:50:56.193470 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:56.193410 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 20:50:56.193470 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:56.193448 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls podName:81bd5151-ce78-4a7c-9ebf-b787d915430c nodeName:}" failed. No retries permitted until 2026-09-09 20:50:56.693435122 +0000 UTC m=+34.154077025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls") pod "dns-default-gb8k5" (UID: "81bd5151-ce78-4a7c-9ebf-b787d915430c") : secret "dns-default-metrics-tls" not found Sep 09 20:50:56.193470 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.193410 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bea0cf71-87de-42ff-bcef-683009062999-cert\") pod \"ingress-canary-77lm8\" (UID: \"bea0cf71-87de-42ff-bcef-683009062999\") " pod="openshift-ingress-canary/ingress-canary-77lm8" Sep 09 20:50:56.193610 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.193482 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/81bd5151-ce78-4a7c-9ebf-b787d915430c-config-volume\") pod \"dns-default-gb8k5\" (UID: \"81bd5151-ce78-4a7c-9ebf-b787d915430c\") " pod="openshift-dns/dns-default-gb8k5" Sep 09 20:50:56.193610 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.193518 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thcm2\" (UniqueName: \"kubernetes.io/projected/81bd5151-ce78-4a7c-9ebf-b787d915430c-kube-api-access-thcm2\") pod \"dns-default-gb8k5\" (UID: \"81bd5151-ce78-4a7c-9ebf-b787d915430c\") " pod="openshift-dns/dns-default-gb8k5" Sep 09 20:50:56.193670 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.193611 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81bd5151-ce78-4a7c-9ebf-b787d915430c-tmp-dir\") pod \"dns-default-gb8k5\" (UID: \"81bd5151-ce78-4a7c-9ebf-b787d915430c\") " pod="openshift-dns/dns-default-gb8k5" Sep 09 20:50:56.193997 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.193977 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/81bd5151-ce78-4a7c-9ebf-b787d915430c-config-volume\") pod \"dns-default-gb8k5\" (UID: \"81bd5151-ce78-4a7c-9ebf-b787d915430c\") " pod="openshift-dns/dns-default-gb8k5" Sep 09 20:50:56.197270 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.197254 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bea0cf71-87de-42ff-bcef-683009062999-cert\") pod \"ingress-canary-77lm8\" (UID: \"bea0cf71-87de-42ff-bcef-683009062999\") " pod="openshift-ingress-canary/ingress-canary-77lm8" Sep 09 20:50:56.203496 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.203471 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thcm2\" (UniqueName: \"kubernetes.io/projected/81bd5151-ce78-4a7c-9ebf-b787d915430c-kube-api-access-thcm2\") pod \"dns-default-gb8k5\" (UID: \"81bd5151-ce78-4a7c-9ebf-b787d915430c\") " pod="openshift-dns/dns-default-gb8k5" Sep 09 20:50:56.203736 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.203721 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-565l8\" (UniqueName: \"kubernetes.io/projected/bea0cf71-87de-42ff-bcef-683009062999-kube-api-access-565l8\") pod \"ingress-canary-77lm8\" (UID: \"bea0cf71-87de-42ff-bcef-683009062999\") " pod="openshift-ingress-canary/ingress-canary-77lm8" Sep 09 20:50:56.281829 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.281790 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-77lm8" Sep 09 20:50:56.304606 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.304581 2580 generic.go:356] "Generic (PLEG): container finished" podID="75366d82-a9b6-4a08-a9cf-a3c9cd38ef85" containerID="e743114d00ffc549737f1cf1428ee28e2a05409de1993efe119f047e5d3525c8" exitCode=0 Sep 09 20:50:56.304692 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.304617 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rwtrs" event={"ID":"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85","Type":"ContainerDied","Data":"e743114d00ffc549737f1cf1428ee28e2a05409de1993efe119f047e5d3525c8"} Sep 09 20:50:56.489535 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.489365 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:50:56.547148 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.547080 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:50:56.606766 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.606744 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-77lm8"] Sep 09 20:50:56.609828 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:50:56.609806 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea0cf71_87de_42ff_bcef_683009062999.slice/crio-b6b42a722d14fd0171d441855a58a20005bb5d120eb9e4ff0447fe3fda757ab9 WatchSource:0}: Error finding container b6b42a722d14fd0171d441855a58a20005bb5d120eb9e4ff0447fe3fda757ab9: Status 404 returned error can't find the container with id b6b42a722d14fd0171d441855a58a20005bb5d120eb9e4ff0447fe3fda757ab9 Sep 09 20:50:56.688693 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.688670 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:50:56.697611 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.697588 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls\") pod \"dns-default-gb8k5\" (UID: \"81bd5151-ce78-4a7c-9ebf-b787d915430c\") " pod="openshift-dns/dns-default-gb8k5" Sep 09 20:50:56.697709 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:56.697697 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 20:50:56.697754 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:56.697739 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls podName:81bd5151-ce78-4a7c-9ebf-b787d915430c nodeName:}" failed. No retries permitted until 2026-09-09 20:50:57.697727719 +0000 UTC m=+35.158369619 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls") pod "dns-default-gb8k5" (UID: "81bd5151-ce78-4a7c-9ebf-b787d915430c") : secret "dns-default-metrics-tls" not found Sep 09 20:50:56.788728 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:56.788703 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:50:57.309798 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:57.309776 2580 generic.go:356] "Generic (PLEG): container finished" podID="75366d82-a9b6-4a08-a9cf-a3c9cd38ef85" containerID="9899250bf626401e15944faab91952b530aae99d9a89dd70eac3efac53f14766" exitCode=0 Sep 09 20:50:57.309928 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:57.309832 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rwtrs" event={"ID":"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85","Type":"ContainerDied","Data":"9899250bf626401e15944faab91952b530aae99d9a89dd70eac3efac53f14766"} Sep 09 20:50:57.311063 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:57.311042 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-77lm8" event={"ID":"bea0cf71-87de-42ff-bcef-683009062999","Type":"ContainerStarted","Data":"b6b42a722d14fd0171d441855a58a20005bb5d120eb9e4ff0447fe3fda757ab9"} Sep 09 20:50:57.703967 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:57.703939 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls\") pod \"dns-default-gb8k5\" (UID: \"81bd5151-ce78-4a7c-9ebf-b787d915430c\") " pod="openshift-dns/dns-default-gb8k5" Sep 09 20:50:57.704610 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:57.704114 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 20:50:57.704610 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:57.704183 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls podName:81bd5151-ce78-4a7c-9ebf-b787d915430c nodeName:}" failed. No retries permitted until 2026-09-09 20:50:59.704167553 +0000 UTC m=+37.164809461 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls") pod "dns-default-gb8k5" (UID: "81bd5151-ce78-4a7c-9ebf-b787d915430c") : secret "dns-default-metrics-tls" not found Sep 09 20:50:58.317696 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:58.317665 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rwtrs" event={"ID":"75366d82-a9b6-4a08-a9cf-a3c9cd38ef85","Type":"ContainerStarted","Data":"cc2107e3b244a99aae1f6e50d7cf3eb6132efe1a078b8e1b2e812e0a6289367f"} Sep 09 20:50:58.344488 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:58.344447 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rwtrs" podStartSLOduration=6.878086661 podStartE2EDuration="35.344433361s" totalImagesPullingTime="28.4663467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:50:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:58.344336135 +0000 UTC m=+35.804978059" watchObservedRunningTime="2026-09-09 20:50:58.344433361 +0000 UTC m=+35.805075274" Sep 09 20:50:58.468553 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:58.468517 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:50:58.538477 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:58.538440 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:50:59.718182 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:50:59.718151 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls\") pod \"dns-default-gb8k5\" (UID: \"81bd5151-ce78-4a7c-9ebf-b787d915430c\") " pod="openshift-dns/dns-default-gb8k5" Sep 09 20:50:59.718552 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:59.718270 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 20:50:59.718552 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:50:59.718322 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls podName:81bd5151-ce78-4a7c-9ebf-b787d915430c nodeName:}" failed. No retries permitted until 2026-09-09 20:51:03.718309666 +0000 UTC m=+41.178951567 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls") pod "dns-default-gb8k5" (UID: "81bd5151-ce78-4a7c-9ebf-b787d915430c") : secret "dns-default-metrics-tls" not found Sep 09 20:51:00.323508 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:00.323451 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-77lm8" event={"ID":"bea0cf71-87de-42ff-bcef-683009062999","Type":"ContainerStarted","Data":"8c456b678f0df6a3749ef515b0b1e28ccfb61b9be6adc6409aa6199e037f28d6"} Sep 09 20:51:00.339471 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:00.339428 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-77lm8" podStartSLOduration=2.705565823 podStartE2EDuration="5.339413263s" totalImagesPullingTime="2.63384744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:51:00.338944092 +0000 UTC m=+37.799586014" watchObservedRunningTime="2026-09-09 20:51:00.339413263 +0000 UTC m=+37.800055185" Sep 09 20:51:00.756624 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:00.756590 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:51:00.854623 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:00.854600 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:51:01.997641 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:01.997609 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:51:02.052482 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:02.052454 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:51:02.421595 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:02.421537 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:51:02.522840 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:02.522815 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:51:03.744460 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:03.744436 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls\") pod \"dns-default-gb8k5\" (UID: \"81bd5151-ce78-4a7c-9ebf-b787d915430c\") " pod="openshift-dns/dns-default-gb8k5" Sep 09 20:51:03.744759 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:51:03.744534 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 20:51:03.744759 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:51:03.744584 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls podName:81bd5151-ce78-4a7c-9ebf-b787d915430c nodeName:}" failed. No retries permitted until 2026-09-09 20:51:11.744571181 +0000 UTC m=+49.205213081 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls") pod "dns-default-gb8k5" (UID: "81bd5151-ce78-4a7c-9ebf-b787d915430c") : secret "dns-default-metrics-tls" not found Sep 09 20:51:11.798465 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:11.798430 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls\") pod \"dns-default-gb8k5\" (UID: \"81bd5151-ce78-4a7c-9ebf-b787d915430c\") " pod="openshift-dns/dns-default-gb8k5" Sep 09 20:51:11.798911 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:51:11.798541 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 20:51:11.798911 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:51:11.798602 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls podName:81bd5151-ce78-4a7c-9ebf-b787d915430c nodeName:}" failed. No retries permitted until 2026-09-09 20:51:27.798586448 +0000 UTC m=+65.259228348 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls") pod "dns-default-gb8k5" (UID: "81bd5151-ce78-4a7c-9ebf-b787d915430c") : secret "dns-default-metrics-tls" not found Sep 09 20:51:17.922551 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:17.922483 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:51:18.023879 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:18.023851 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:51:22.304701 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:22.304671 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tctqv" Sep 09 20:51:22.459329 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:22.459296 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:51:22.515105 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:22.515080 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:51:23.954863 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:23.954830 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:51:24.055921 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:24.055871 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:51:27.805438 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:27.805402 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs\") pod \"network-metrics-daemon-9dfd9\" (UID: \"f8e8b1d2-6523-403e-bace-b96b38d8449b\") " pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:51:27.805438 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:27.805440 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls\") pod \"dns-default-gb8k5\" (UID: \"81bd5151-ce78-4a7c-9ebf-b787d915430c\") " pod="openshift-dns/dns-default-gb8k5" Sep 09 20:51:27.805815 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:27.805461 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91f58455-46ed-4617-bfe6-ef79a8fc05f3-original-pull-secret\") pod \"global-pull-secret-syncer-csblg\" (UID: \"91f58455-46ed-4617-bfe6-ef79a8fc05f3\") " pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:51:27.805815 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:51:27.805543 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 20:51:27.805815 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:51:27.805598 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls podName:81bd5151-ce78-4a7c-9ebf-b787d915430c nodeName:}" failed. No retries permitted until 2026-09-09 20:51:59.805583915 +0000 UTC m=+97.266225837 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls") pod "dns-default-gb8k5" (UID: "81bd5151-ce78-4a7c-9ebf-b787d915430c") : secret "dns-default-metrics-tls" not found Sep 09 20:51:27.807583 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:27.807568 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 20:51:27.807646 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:27.807604 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 20:51:27.816352 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:51:27.816337 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 20:51:27.816406 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:51:27.816395 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs podName:f8e8b1d2-6523-403e-bace-b96b38d8449b nodeName:}" failed. No retries permitted until 2026-09-09 20:52:31.816379849 +0000 UTC m=+129.277021766 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs") pod "network-metrics-daemon-9dfd9" (UID: "f8e8b1d2-6523-403e-bace-b96b38d8449b") : secret "metrics-daemon-secret" not found Sep 09 20:51:27.819366 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:27.819347 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91f58455-46ed-4617-bfe6-ef79a8fc05f3-original-pull-secret\") pod \"global-pull-secret-syncer-csblg\" (UID: \"91f58455-46ed-4617-bfe6-ef79a8fc05f3\") " pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:51:27.870757 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:27.870741 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-csblg" Sep 09 20:51:27.906397 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:27.906361 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzjcn\" (UniqueName: \"kubernetes.io/projected/b5e2d5ce-182b-46ef-b30d-52e8115c4f54-kube-api-access-bzjcn\") pod \"network-check-target-tbqnj\" (UID: \"b5e2d5ce-182b-46ef-b30d-52e8115c4f54\") " pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:51:27.909348 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:27.909331 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 20:51:27.918819 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:27.918798 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 20:51:27.930076 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:27.930049 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzjcn\" (UniqueName: \"kubernetes.io/projected/b5e2d5ce-182b-46ef-b30d-52e8115c4f54-kube-api-access-bzjcn\") pod \"network-check-target-tbqnj\" (UID: \"b5e2d5ce-182b-46ef-b30d-52e8115c4f54\") " pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:51:28.136617 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:28.136558 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-csblg"] Sep 09 20:51:28.139756 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:51:28.139727 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f58455_46ed_4617_bfe6_ef79a8fc05f3.slice/crio-21db1a381858121b578cb9281db951d75254e3b951ceef6148a88ea48feb0105 WatchSource:0}: Error finding container 21db1a381858121b578cb9281db951d75254e3b951ceef6148a88ea48feb0105: Status 404 returned error can't find the container with id 21db1a381858121b578cb9281db951d75254e3b951ceef6148a88ea48feb0105 Sep 09 20:51:28.177133 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:28.177114 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wskns\"" Sep 09 20:51:28.184849 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:28.184836 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:51:28.380821 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:28.380787 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-csblg" event={"ID":"91f58455-46ed-4617-bfe6-ef79a8fc05f3","Type":"ContainerStarted","Data":"21db1a381858121b578cb9281db951d75254e3b951ceef6148a88ea48feb0105"} Sep 09 20:51:28.448303 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:28.448282 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tbqnj"] Sep 09 20:51:28.451135 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:51:28.451107 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e2d5ce_182b_46ef_b30d_52e8115c4f54.slice/crio-1ca526cb2d6cd8e3796227a2749ab66aca965a55453747ff16bc3868d68707b9 WatchSource:0}: Error finding container 1ca526cb2d6cd8e3796227a2749ab66aca965a55453747ff16bc3868d68707b9: Status 404 returned error can't find the container with id 1ca526cb2d6cd8e3796227a2749ab66aca965a55453747ff16bc3868d68707b9 Sep 09 20:51:29.385356 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:29.385259 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tbqnj" event={"ID":"b5e2d5ce-182b-46ef-b30d-52e8115c4f54","Type":"ContainerStarted","Data":"1ca526cb2d6cd8e3796227a2749ab66aca965a55453747ff16bc3868d68707b9"} Sep 09 20:51:31.258794 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:31.258753 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:51:31.359620 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:31.359531 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:51:33.396683 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:33.396598 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-csblg" event={"ID":"91f58455-46ed-4617-bfe6-ef79a8fc05f3","Type":"ContainerStarted","Data":"ad76e0f0d48f14998d4d8283600f402c0fd510b3044f8336f03679e1e596d725"} Sep 09 20:51:33.398090 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:33.398068 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tbqnj" event={"ID":"b5e2d5ce-182b-46ef-b30d-52e8115c4f54","Type":"ContainerStarted","Data":"f73bfda0ba9793fd8ca9e692e8979a3cf08d8a917b494733bc58ffe53301f229"} Sep 09 20:51:33.398212 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:33.398194 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:51:33.410331 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:33.410292 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-csblg" podStartSLOduration=65.536667733 podStartE2EDuration="1m10.410275633s" totalImagesPullingTime="4.8736079s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:51:33.40986752 +0000 UTC m=+70.870509442" watchObservedRunningTime="2026-09-09 20:51:33.410275633 +0000 UTC m=+70.870917556" Sep 09 20:51:33.434487 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:33.434444 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tbqnj" podStartSLOduration=65.82234723 podStartE2EDuration="1m10.434430503s" totalImagesPullingTime="4.612083273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:51:33.433800344 +0000 UTC m=+70.894442265" watchObservedRunningTime="2026-09-09 20:51:33.434430503 +0000 UTC m=+70.895072426" Sep 09 20:51:49.842127 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:49.842018 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-65fdb6884b-rz9ss"] Sep 09 20:51:49.845983 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:49.845966 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65fdb6884b-rz9ss" Sep 09 20:51:49.848117 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:49.848094 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 09 20:51:49.848939 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:49.848917 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 09 20:51:49.849031 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:49.848970 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 09 20:51:49.849031 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:49.849009 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 09 20:51:49.856521 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:49.856500 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-65fdb6884b-rz9ss"] Sep 09 20:51:49.874122 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:49.874102 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt"] Sep 09 20:51:49.876737 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:49.876724 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" Sep 09 20:51:49.878836 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:49.878814 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 09 20:51:49.878978 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:49.878839 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 09 20:51:49.878978 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:49.878844 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 09 20:51:49.878978 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:49.878873 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 09 20:51:49.886865 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:49.886824 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt"] Sep 09 20:51:49.932745 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:49.932721 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/05c7f884-449b-4624-8a43-4e464d2aa55e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5d6d8f9846-ncmmt\" (UID: \"05c7f884-449b-4624-8a43-4e464d2aa55e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" Sep 09 20:51:49.932839 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:49.932746 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6c5d536d-2c42-40a8-9833-ed3b7dfd3f3d-klusterlet-config\") pod \"klusterlet-addon-workmgr-65fdb6884b-rz9ss\" (UID: \"6c5d536d-2c42-40a8-9833-ed3b7dfd3f3d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65fdb6884b-rz9ss" Sep 09 20:51:49.932839 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:49.932762 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c5d536d-2c42-40a8-9833-ed3b7dfd3f3d-tmp\") pod \"klusterlet-addon-workmgr-65fdb6884b-rz9ss\" (UID: \"6c5d536d-2c42-40a8-9833-ed3b7dfd3f3d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65fdb6884b-rz9ss" Sep 09 20:51:49.932839 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:49.932780 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/05c7f884-449b-4624-8a43-4e464d2aa55e-hub\") pod \"cluster-proxy-proxy-agent-5d6d8f9846-ncmmt\" (UID: \"05c7f884-449b-4624-8a43-4e464d2aa55e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" Sep 09 20:51:49.933008 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:49.932867 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/05c7f884-449b-4624-8a43-4e464d2aa55e-ca\") pod \"cluster-proxy-proxy-agent-5d6d8f9846-ncmmt\" (UID: \"05c7f884-449b-4624-8a43-4e464d2aa55e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" Sep 09 20:51:49.933008 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:49.932914 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zdlg\" (UniqueName: \"kubernetes.io/projected/6c5d536d-2c42-40a8-9833-ed3b7dfd3f3d-kube-api-access-6zdlg\") pod \"klusterlet-addon-workmgr-65fdb6884b-rz9ss\" (UID: \"6c5d536d-2c42-40a8-9833-ed3b7dfd3f3d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65fdb6884b-rz9ss" Sep 09 20:51:49.933008 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:49.932940 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/05c7f884-449b-4624-8a43-4e464d2aa55e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5d6d8f9846-ncmmt\" (UID: \"05c7f884-449b-4624-8a43-4e464d2aa55e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" Sep 09 20:51:49.933008 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:49.932976 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/05c7f884-449b-4624-8a43-4e464d2aa55e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5d6d8f9846-ncmmt\" (UID: \"05c7f884-449b-4624-8a43-4e464d2aa55e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" Sep 09 20:51:49.933151 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:49.933017 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsgcg\" (UniqueName: \"kubernetes.io/projected/05c7f884-449b-4624-8a43-4e464d2aa55e-kube-api-access-lsgcg\") pod \"cluster-proxy-proxy-agent-5d6d8f9846-ncmmt\" (UID: \"05c7f884-449b-4624-8a43-4e464d2aa55e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" Sep 09 20:51:50.034161 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:50.034142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/05c7f884-449b-4624-8a43-4e464d2aa55e-ca\") pod \"cluster-proxy-proxy-agent-5d6d8f9846-ncmmt\" (UID: \"05c7f884-449b-4624-8a43-4e464d2aa55e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" Sep 09 20:51:50.034232 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:50.034168 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zdlg\" (UniqueName: \"kubernetes.io/projected/6c5d536d-2c42-40a8-9833-ed3b7dfd3f3d-kube-api-access-6zdlg\") pod \"klusterlet-addon-workmgr-65fdb6884b-rz9ss\" (UID: \"6c5d536d-2c42-40a8-9833-ed3b7dfd3f3d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65fdb6884b-rz9ss" Sep 09 20:51:50.034232 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:50.034189 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/05c7f884-449b-4624-8a43-4e464d2aa55e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5d6d8f9846-ncmmt\" (UID: \"05c7f884-449b-4624-8a43-4e464d2aa55e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" Sep 09 20:51:50.034232 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:50.034205 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/05c7f884-449b-4624-8a43-4e464d2aa55e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5d6d8f9846-ncmmt\" (UID: \"05c7f884-449b-4624-8a43-4e464d2aa55e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" Sep 09 20:51:50.034232 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:50.034219 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsgcg\" (UniqueName: \"kubernetes.io/projected/05c7f884-449b-4624-8a43-4e464d2aa55e-kube-api-access-lsgcg\") pod \"cluster-proxy-proxy-agent-5d6d8f9846-ncmmt\" (UID: \"05c7f884-449b-4624-8a43-4e464d2aa55e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" Sep 09 20:51:50.034413 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:50.034235 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/05c7f884-449b-4624-8a43-4e464d2aa55e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5d6d8f9846-ncmmt\" (UID: \"05c7f884-449b-4624-8a43-4e464d2aa55e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" Sep 09 20:51:50.034413 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:50.034261 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6c5d536d-2c42-40a8-9833-ed3b7dfd3f3d-klusterlet-config\") pod \"klusterlet-addon-workmgr-65fdb6884b-rz9ss\" (UID: \"6c5d536d-2c42-40a8-9833-ed3b7dfd3f3d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65fdb6884b-rz9ss" Sep 09 20:51:50.034413 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:50.034371 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c5d536d-2c42-40a8-9833-ed3b7dfd3f3d-tmp\") pod \"klusterlet-addon-workmgr-65fdb6884b-rz9ss\" (UID: \"6c5d536d-2c42-40a8-9833-ed3b7dfd3f3d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65fdb6884b-rz9ss" Sep 09 20:51:50.034413 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:50.034400 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/05c7f884-449b-4624-8a43-4e464d2aa55e-hub\") pod \"cluster-proxy-proxy-agent-5d6d8f9846-ncmmt\" (UID: \"05c7f884-449b-4624-8a43-4e464d2aa55e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" Sep 09 20:51:50.034728 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:50.034700 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c5d536d-2c42-40a8-9833-ed3b7dfd3f3d-tmp\") pod \"klusterlet-addon-workmgr-65fdb6884b-rz9ss\" (UID: \"6c5d536d-2c42-40a8-9833-ed3b7dfd3f3d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65fdb6884b-rz9ss" Sep 09 20:51:50.035056 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:50.034940 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/05c7f884-449b-4624-8a43-4e464d2aa55e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5d6d8f9846-ncmmt\" (UID: \"05c7f884-449b-4624-8a43-4e464d2aa55e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" Sep 09 20:51:50.037205 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:50.037184 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/05c7f884-449b-4624-8a43-4e464d2aa55e-ca\") pod \"cluster-proxy-proxy-agent-5d6d8f9846-ncmmt\" (UID: \"05c7f884-449b-4624-8a43-4e464d2aa55e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" Sep 09 20:51:50.037306 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:50.037225 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/05c7f884-449b-4624-8a43-4e464d2aa55e-hub\") pod \"cluster-proxy-proxy-agent-5d6d8f9846-ncmmt\" (UID: \"05c7f884-449b-4624-8a43-4e464d2aa55e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" Sep 09 20:51:50.037306 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:50.037255 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/05c7f884-449b-4624-8a43-4e464d2aa55e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5d6d8f9846-ncmmt\" (UID: \"05c7f884-449b-4624-8a43-4e464d2aa55e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" Sep 09 20:51:50.037306 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:50.037284 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6c5d536d-2c42-40a8-9833-ed3b7dfd3f3d-klusterlet-config\") pod \"klusterlet-addon-workmgr-65fdb6884b-rz9ss\" (UID: \"6c5d536d-2c42-40a8-9833-ed3b7dfd3f3d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65fdb6884b-rz9ss" Sep 09 20:51:50.037411 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:50.037380 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/05c7f884-449b-4624-8a43-4e464d2aa55e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5d6d8f9846-ncmmt\" (UID: \"05c7f884-449b-4624-8a43-4e464d2aa55e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" Sep 09 20:51:50.042621 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:50.042598 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsgcg\" (UniqueName: \"kubernetes.io/projected/05c7f884-449b-4624-8a43-4e464d2aa55e-kube-api-access-lsgcg\") pod \"cluster-proxy-proxy-agent-5d6d8f9846-ncmmt\" (UID: \"05c7f884-449b-4624-8a43-4e464d2aa55e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" Sep 09 20:51:50.042715 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:50.042695 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zdlg\" (UniqueName: \"kubernetes.io/projected/6c5d536d-2c42-40a8-9833-ed3b7dfd3f3d-kube-api-access-6zdlg\") pod \"klusterlet-addon-workmgr-65fdb6884b-rz9ss\" (UID: \"6c5d536d-2c42-40a8-9833-ed3b7dfd3f3d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65fdb6884b-rz9ss" Sep 09 20:51:50.154906 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:50.154840 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65fdb6884b-rz9ss" Sep 09 20:51:50.195656 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:50.195628 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" Sep 09 20:51:50.420876 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:50.420816 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-65fdb6884b-rz9ss"] Sep 09 20:51:50.423478 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:51:50.423450 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c5d536d_2c42_40a8_9833_ed3b7dfd3f3d.slice/crio-17db0aaea04e2d72fbca7e0c463c117193b2d2c1cec58b00b34af1e9a1d7b70e WatchSource:0}: Error finding container 17db0aaea04e2d72fbca7e0c463c117193b2d2c1cec58b00b34af1e9a1d7b70e: Status 404 returned error can't find the container with id 17db0aaea04e2d72fbca7e0c463c117193b2d2c1cec58b00b34af1e9a1d7b70e Sep 09 20:51:50.432645 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:50.432621 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65fdb6884b-rz9ss" event={"ID":"6c5d536d-2c42-40a8-9833-ed3b7dfd3f3d","Type":"ContainerStarted","Data":"17db0aaea04e2d72fbca7e0c463c117193b2d2c1cec58b00b34af1e9a1d7b70e"} Sep 09 20:51:50.469996 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:50.469976 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt"] Sep 09 20:51:50.472489 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:51:50.472470 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05c7f884_449b_4624_8a43_4e464d2aa55e.slice/crio-d8251c2389eebcbe93037bab989ea276bbccd0aac511fe5791632a172dac5f87 WatchSource:0}: Error finding container d8251c2389eebcbe93037bab989ea276bbccd0aac511fe5791632a172dac5f87: Status 404 returned error can't find the container with id d8251c2389eebcbe93037bab989ea276bbccd0aac511fe5791632a172dac5f87 Sep 09 20:51:51.438342 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:51.438279 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" event={"ID":"05c7f884-449b-4624-8a43-4e464d2aa55e","Type":"ContainerStarted","Data":"d8251c2389eebcbe93037bab989ea276bbccd0aac511fe5791632a172dac5f87"} Sep 09 20:51:56.454697 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:56.454661 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" event={"ID":"05c7f884-449b-4624-8a43-4e464d2aa55e","Type":"ContainerStarted","Data":"1bad55293f78f403ab2733c9710ba1721c5128cbb6e07204b4ac49b31ee630ca"} Sep 09 20:51:56.455065 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:56.454706 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" event={"ID":"05c7f884-449b-4624-8a43-4e464d2aa55e","Type":"ContainerStarted","Data":"4eece6a24e79417b83ac1e8bc612a18d1c48e5c60ff1f3b04ccc3c2d189a7960"} Sep 09 20:51:56.455065 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:56.454718 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" event={"ID":"05c7f884-449b-4624-8a43-4e464d2aa55e","Type":"ContainerStarted","Data":"1bba438aabb03b00ecd1e4aee74fa9d0d8a4544b0f4c1b3006267189d22a4751"} Sep 09 20:51:56.456131 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:56.456107 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65fdb6884b-rz9ss" event={"ID":"6c5d536d-2c42-40a8-9833-ed3b7dfd3f3d","Type":"ContainerStarted","Data":"2f468ed4aa162f23059a6a3e38238275271a9209ee33923ce4facd441bb6a891"} Sep 09 20:51:56.456291 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:56.456276 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65fdb6884b-rz9ss" Sep 09 20:51:56.457633 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:56.457617 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65fdb6884b-rz9ss" Sep 09 20:51:56.472765 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:56.472704 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" podStartSLOduration=1.900161268 podStartE2EDuration="7.472686713s" totalImagesPullingTime="5.572525445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:51:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:51:56.471766039 +0000 UTC m=+93.932407959" watchObservedRunningTime="2026-09-09 20:51:56.472686713 +0000 UTC m=+93.933328640" Sep 09 20:51:56.487014 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:56.486975 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65fdb6884b-rz9ss" podStartSLOduration=1.863066369 podStartE2EDuration="7.486961938s" totalImagesPullingTime="5.623895569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:51:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:51:56.486645739 +0000 UTC m=+93.947287663" watchObservedRunningTime="2026-09-09 20:51:56.486961938 +0000 UTC m=+93.947603859" Sep 09 20:51:59.899156 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:51:59.899123 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls\") pod \"dns-default-gb8k5\" (UID: \"81bd5151-ce78-4a7c-9ebf-b787d915430c\") " pod="openshift-dns/dns-default-gb8k5" Sep 09 20:51:59.899498 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:51:59.899220 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 20:51:59.899498 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:51:59.899275 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls podName:81bd5151-ce78-4a7c-9ebf-b787d915430c nodeName:}" failed. No retries permitted until 2026-09-09 20:53:03.899258811 +0000 UTC m=+161.359900733 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls") pod "dns-default-gb8k5" (UID: "81bd5151-ce78-4a7c-9ebf-b787d915430c") : secret "dns-default-metrics-tls" not found Sep 09 20:52:04.402480 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:52:04.402454 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tbqnj" Sep 09 20:52:05.336595 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:52:05.336541 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:31.906082 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:52:31.906044 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs\") pod \"network-metrics-daemon-9dfd9\" (UID: \"f8e8b1d2-6523-403e-bace-b96b38d8449b\") " pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:52:31.906524 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:52:31.906160 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 20:52:31.906524 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:52:31.906219 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs podName:f8e8b1d2-6523-403e-bace-b96b38d8449b nodeName:}" failed. No retries permitted until 2026-09-09 20:54:33.906205196 +0000 UTC m=+251.366847101 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs") pod "network-metrics-daemon-9dfd9" (UID: "f8e8b1d2-6523-403e-bace-b96b38d8449b") : secret "metrics-daemon-secret" not found Sep 09 20:52:58.967325 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:52:58.967276 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-gb8k5" podUID="81bd5151-ce78-4a7c-9ebf-b787d915430c" Sep 09 20:52:59.064560 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:52:59.064526 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-9dfd9" podUID="f8e8b1d2-6523-403e-bace-b96b38d8449b" Sep 09 20:52:59.622660 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:52:59.622636 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gb8k5" Sep 09 20:53:03.904121 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:53:03.904082 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls\") pod \"dns-default-gb8k5\" (UID: \"81bd5151-ce78-4a7c-9ebf-b787d915430c\") " pod="openshift-dns/dns-default-gb8k5" Sep 09 20:53:03.904548 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:53:03.904176 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 20:53:03.904548 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:53:03.904223 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls podName:81bd5151-ce78-4a7c-9ebf-b787d915430c nodeName:}" failed. No retries permitted until 2026-09-09 20:55:05.904210583 +0000 UTC m=+283.364852483 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls") pod "dns-default-gb8k5" (UID: "81bd5151-ce78-4a7c-9ebf-b787d915430c") : secret "dns-default-metrics-tls" not found Sep 09 20:53:10.037368 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:53:10.037340 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:53:27.892370 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:53:27.892339 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:56.456971 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:53:56.456932 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65fdb6884b-rz9ss" podUID="6c5d536d-2c42-40a8-9833-ed3b7dfd3f3d" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.8:8000/readyz\": dial tcp 10.133.0.8:8000: connect: connection refused" Sep 09 20:53:56.767108 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:53:56.767055 2580 generic.go:356] "Generic (PLEG): container finished" podID="6c5d536d-2c42-40a8-9833-ed3b7dfd3f3d" containerID="2f468ed4aa162f23059a6a3e38238275271a9209ee33923ce4facd441bb6a891" exitCode=1 Sep 09 20:53:56.767108 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:53:56.767105 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65fdb6884b-rz9ss" event={"ID":"6c5d536d-2c42-40a8-9833-ed3b7dfd3f3d","Type":"ContainerDied","Data":"2f468ed4aa162f23059a6a3e38238275271a9209ee33923ce4facd441bb6a891"} Sep 09 20:53:56.767359 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:53:56.767347 2580 scope.go:117] "RemoveContainer" containerID="2f468ed4aa162f23059a6a3e38238275271a9209ee33923ce4facd441bb6a891" Sep 09 20:53:57.771863 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:53:57.771834 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65fdb6884b-rz9ss" event={"ID":"6c5d536d-2c42-40a8-9833-ed3b7dfd3f3d","Type":"ContainerStarted","Data":"f0e8ae0b840ef94b1dcb91ebcf45faa6db12bef19c6f65d254af30572dc026c0"} Sep 09 20:53:57.772256 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:53:57.772211 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65fdb6884b-rz9ss" Sep 09 20:53:57.772558 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:53:57.772542 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65fdb6884b-rz9ss" Sep 09 20:54:19.607037 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:19.606995 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:33.967490 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:33.967453 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs\") pod \"network-metrics-daemon-9dfd9\" (UID: \"f8e8b1d2-6523-403e-bace-b96b38d8449b\") " pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:54:33.967949 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:54:33.967556 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 20:54:33.967949 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:54:33.967605 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs podName:f8e8b1d2-6523-403e-bace-b96b38d8449b nodeName:}" failed. No retries permitted until 2026-09-09 20:56:35.967591483 +0000 UTC m=+373.428233388 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs") pod "network-metrics-daemon-9dfd9" (UID: "f8e8b1d2-6523-403e-bace-b96b38d8449b") : secret "metrics-daemon-secret" not found Sep 09 20:54:44.120662 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:44.120623 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:44.176307 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:44.176282 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:45.991153 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:45.991118 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:46.055095 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:46.055070 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:48.043415 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.043383 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:48.120546 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.120519 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:48.453455 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.453427 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:48.567144 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.567116 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:48.606332 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.606311 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6pjzl"] Sep 09 20:54:48.609214 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.609201 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:54:48.612981 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.612963 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 09 20:54:48.613054 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.612992 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9gf4v\"" Sep 09 20:54:48.613247 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.613235 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 09 20:54:48.613450 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.613434 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 09 20:54:48.613513 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.613496 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 09 20:54:48.626505 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.626484 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6pjzl"] Sep 09 20:54:48.660070 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.660041 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13b9d045-0f29-42c6-b394-53a075824ec8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6pjzl\" (UID: \"13b9d045-0f29-42c6-b394-53a075824ec8\") " pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:54:48.660152 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.660093 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13b9d045-0f29-42c6-b394-53a075824ec8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pjzl\" (UID: \"13b9d045-0f29-42c6-b394-53a075824ec8\") " pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:54:48.660152 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.660119 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6qgk\" (UniqueName: \"kubernetes.io/projected/13b9d045-0f29-42c6-b394-53a075824ec8-kube-api-access-j6qgk\") pod \"insights-runtime-extractor-6pjzl\" (UID: \"13b9d045-0f29-42c6-b394-53a075824ec8\") " pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:54:48.660223 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.660166 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/13b9d045-0f29-42c6-b394-53a075824ec8-crio-socket\") pod \"insights-runtime-extractor-6pjzl\" (UID: \"13b9d045-0f29-42c6-b394-53a075824ec8\") " pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:54:48.660255 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.660219 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/13b9d045-0f29-42c6-b394-53a075824ec8-data-volume\") pod \"insights-runtime-extractor-6pjzl\" (UID: \"13b9d045-0f29-42c6-b394-53a075824ec8\") " pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:54:48.669694 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.669675 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:48.760968 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.760923 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13b9d045-0f29-42c6-b394-53a075824ec8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6pjzl\" (UID: \"13b9d045-0f29-42c6-b394-53a075824ec8\") " pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:54:48.761071 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.760971 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13b9d045-0f29-42c6-b394-53a075824ec8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pjzl\" (UID: \"13b9d045-0f29-42c6-b394-53a075824ec8\") " pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:54:48.761071 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.760989 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6qgk\" (UniqueName: \"kubernetes.io/projected/13b9d045-0f29-42c6-b394-53a075824ec8-kube-api-access-j6qgk\") pod \"insights-runtime-extractor-6pjzl\" (UID: \"13b9d045-0f29-42c6-b394-53a075824ec8\") " pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:54:48.761071 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.761005 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/13b9d045-0f29-42c6-b394-53a075824ec8-crio-socket\") pod \"insights-runtime-extractor-6pjzl\" (UID: \"13b9d045-0f29-42c6-b394-53a075824ec8\") " pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:54:48.761071 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.761027 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/13b9d045-0f29-42c6-b394-53a075824ec8-data-volume\") pod \"insights-runtime-extractor-6pjzl\" (UID: \"13b9d045-0f29-42c6-b394-53a075824ec8\") " pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:54:48.761071 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:54:48.761052 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 20:54:48.761301 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:54:48.761073 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 20:54:48.761301 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.761093 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/13b9d045-0f29-42c6-b394-53a075824ec8-crio-socket\") pod \"insights-runtime-extractor-6pjzl\" (UID: \"13b9d045-0f29-42c6-b394-53a075824ec8\") " pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:54:48.761301 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:54:48.761129 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13b9d045-0f29-42c6-b394-53a075824ec8-kube-rbac-proxy-cm podName:13b9d045-0f29-42c6-b394-53a075824ec8 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:49.261112897 +0000 UTC m=+266.721754801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13b9d045-0f29-42c6-b394-53a075824ec8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6pjzl" (UID: "13b9d045-0f29-42c6-b394-53a075824ec8") : configmap "kube-rbac-proxy" not found Sep 09 20:54:48.761301 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:54:48.761147 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13b9d045-0f29-42c6-b394-53a075824ec8-insights-runtime-extractor-tls podName:13b9d045-0f29-42c6-b394-53a075824ec8 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:49.2611389 +0000 UTC m=+266.721780832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13b9d045-0f29-42c6-b394-53a075824ec8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6pjzl" (UID: "13b9d045-0f29-42c6-b394-53a075824ec8") : secret "insights-runtime-extractor-tls" not found Sep 09 20:54:48.761301 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.761288 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/13b9d045-0f29-42c6-b394-53a075824ec8-data-volume\") pod \"insights-runtime-extractor-6pjzl\" (UID: \"13b9d045-0f29-42c6-b394-53a075824ec8\") " pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:54:48.773231 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.773207 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6qgk\" (UniqueName: \"kubernetes.io/projected/13b9d045-0f29-42c6-b394-53a075824ec8-kube-api-access-j6qgk\") pod \"insights-runtime-extractor-6pjzl\" (UID: \"13b9d045-0f29-42c6-b394-53a075824ec8\") " pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:54:48.854146 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:48.854121 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:49.264288 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:49.264261 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13b9d045-0f29-42c6-b394-53a075824ec8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6pjzl\" (UID: \"13b9d045-0f29-42c6-b394-53a075824ec8\") " pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:54:49.264651 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:49.264311 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13b9d045-0f29-42c6-b394-53a075824ec8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pjzl\" (UID: \"13b9d045-0f29-42c6-b394-53a075824ec8\") " pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:54:49.264651 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:54:49.264377 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 20:54:49.264651 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:54:49.264388 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 20:54:49.264651 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:54:49.264424 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13b9d045-0f29-42c6-b394-53a075824ec8-kube-rbac-proxy-cm podName:13b9d045-0f29-42c6-b394-53a075824ec8 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:50.264410423 +0000 UTC m=+267.725052326 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13b9d045-0f29-42c6-b394-53a075824ec8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6pjzl" (UID: "13b9d045-0f29-42c6-b394-53a075824ec8") : configmap "kube-rbac-proxy" not found Sep 09 20:54:49.264651 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:54:49.264438 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13b9d045-0f29-42c6-b394-53a075824ec8-insights-runtime-extractor-tls podName:13b9d045-0f29-42c6-b394-53a075824ec8 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:50.264432377 +0000 UTC m=+267.725074278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13b9d045-0f29-42c6-b394-53a075824ec8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6pjzl" (UID: "13b9d045-0f29-42c6-b394-53a075824ec8") : secret "insights-runtime-extractor-tls" not found Sep 09 20:54:49.560014 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:49.559941 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:50.269962 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:50.269934 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13b9d045-0f29-42c6-b394-53a075824ec8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6pjzl\" (UID: \"13b9d045-0f29-42c6-b394-53a075824ec8\") " pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:54:50.270289 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:50.269984 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13b9d045-0f29-42c6-b394-53a075824ec8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pjzl\" (UID: \"13b9d045-0f29-42c6-b394-53a075824ec8\") " pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:54:50.270289 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:54:50.270037 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 20:54:50.270289 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:54:50.270071 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 20:54:50.270289 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:54:50.270095 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13b9d045-0f29-42c6-b394-53a075824ec8-kube-rbac-proxy-cm podName:13b9d045-0f29-42c6-b394-53a075824ec8 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:52.270081306 +0000 UTC m=+269.730723208 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13b9d045-0f29-42c6-b394-53a075824ec8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6pjzl" (UID: "13b9d045-0f29-42c6-b394-53a075824ec8") : configmap "kube-rbac-proxy" not found Sep 09 20:54:50.270289 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:54:50.270113 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13b9d045-0f29-42c6-b394-53a075824ec8-insights-runtime-extractor-tls podName:13b9d045-0f29-42c6-b394-53a075824ec8 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:52.270102371 +0000 UTC m=+269.730744271 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13b9d045-0f29-42c6-b394-53a075824ec8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6pjzl" (UID: "13b9d045-0f29-42c6-b394-53a075824ec8") : secret "insights-runtime-extractor-tls" not found Sep 09 20:54:50.382654 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:50.382626 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:50.459804 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:50.459778 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:50.537416 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:50.537365 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:50.638512 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:50.638491 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:51.332116 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:51.332095 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bxfjt_717799e6-8e3b-4a16-88d6-384d950d5385/dns-node-resolver/0.log" Sep 09 20:54:51.345436 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:51.345409 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:52.129783 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:52.129766 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gx8lm_e64d1b19-5ae3-4758-8753-e83a736855f0/node-ca/0.log" Sep 09 20:54:52.283071 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:52.283049 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13b9d045-0f29-42c6-b394-53a075824ec8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6pjzl\" (UID: \"13b9d045-0f29-42c6-b394-53a075824ec8\") " pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:54:52.283171 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:52.283111 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13b9d045-0f29-42c6-b394-53a075824ec8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pjzl\" (UID: \"13b9d045-0f29-42c6-b394-53a075824ec8\") " pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:54:52.283223 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:54:52.283169 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 20:54:52.283223 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:54:52.283193 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 20:54:52.283296 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:54:52.283225 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13b9d045-0f29-42c6-b394-53a075824ec8-kube-rbac-proxy-cm podName:13b9d045-0f29-42c6-b394-53a075824ec8 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:56.283212251 +0000 UTC m=+273.743854151 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13b9d045-0f29-42c6-b394-53a075824ec8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6pjzl" (UID: "13b9d045-0f29-42c6-b394-53a075824ec8") : configmap "kube-rbac-proxy" not found Sep 09 20:54:52.283296 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:54:52.283239 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13b9d045-0f29-42c6-b394-53a075824ec8-insights-runtime-extractor-tls podName:13b9d045-0f29-42c6-b394-53a075824ec8 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:56.283233028 +0000 UTC m=+273.743874928 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13b9d045-0f29-42c6-b394-53a075824ec8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6pjzl" (UID: "13b9d045-0f29-42c6-b394-53a075824ec8") : secret "insights-runtime-extractor-tls" not found Sep 09 20:54:52.929813 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:52.929788 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-77lm8_bea0cf71-87de-42ff-bcef-683009062999/serve-healthcheck-canary/0.log" Sep 09 20:54:54.304575 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:54.304546 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:55.865599 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:55.865569 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:56.168707 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:56.168637 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:56.308298 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:56.308270 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13b9d045-0f29-42c6-b394-53a075824ec8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pjzl\" (UID: \"13b9d045-0f29-42c6-b394-53a075824ec8\") " pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:54:56.308396 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:56.308319 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13b9d045-0f29-42c6-b394-53a075824ec8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6pjzl\" (UID: \"13b9d045-0f29-42c6-b394-53a075824ec8\") " pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:54:56.308433 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:54:56.308398 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 20:54:56.308433 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:54:56.308398 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 20:54:56.308494 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:54:56.308453 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13b9d045-0f29-42c6-b394-53a075824ec8-kube-rbac-proxy-cm podName:13b9d045-0f29-42c6-b394-53a075824ec8 nodeName:}" failed. No retries permitted until 2026-09-09 20:55:04.308441014 +0000 UTC m=+281.769082919 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13b9d045-0f29-42c6-b394-53a075824ec8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6pjzl" (UID: "13b9d045-0f29-42c6-b394-53a075824ec8") : configmap "kube-rbac-proxy" not found Sep 09 20:54:56.308494 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:54:56.308465 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13b9d045-0f29-42c6-b394-53a075824ec8-insights-runtime-extractor-tls podName:13b9d045-0f29-42c6-b394-53a075824ec8 nodeName:}" failed. No retries permitted until 2026-09-09 20:55:04.308459471 +0000 UTC m=+281.769101371 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13b9d045-0f29-42c6-b394-53a075824ec8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6pjzl" (UID: "13b9d045-0f29-42c6-b394-53a075824ec8") : secret "insights-runtime-extractor-tls" not found Sep 09 20:54:56.554718 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:56.554694 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:56.951960 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:56.951932 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:57.352063 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:57.352018 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:57.561481 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:57.561458 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:58.142879 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:58.142855 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:58.213337 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:58.213316 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:58.605370 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:58.605344 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:58.757113 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:58.757094 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:59.147861 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:59.147834 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:59.957505 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:54:59.957470 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:00.154246 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:00.154214 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:00.557370 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:00.557345 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:00.758060 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:00.758036 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:00.865042 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:00.864976 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:01.173720 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:01.173661 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:01.700704 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:01.700674 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:02.517817 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:02.517785 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:02.623131 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:55:02.623101 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-gb8k5" podUID="81bd5151-ce78-4a7c-9ebf-b787d915430c" Sep 09 20:55:02.729433 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:02.729405 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:02.946026 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:02.945994 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:02.947654 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:02.947628 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gb8k5" Sep 09 20:55:03.142111 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:03.142077 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:03.755210 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:03.755182 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:04.174574 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:04.174466 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:04.318112 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:04.318083 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:04.360643 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:04.360614 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13b9d045-0f29-42c6-b394-53a075824ec8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6pjzl\" (UID: \"13b9d045-0f29-42c6-b394-53a075824ec8\") " pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:55:04.360798 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:04.360669 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13b9d045-0f29-42c6-b394-53a075824ec8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pjzl\" (UID: \"13b9d045-0f29-42c6-b394-53a075824ec8\") " pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:55:04.361164 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:04.361145 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13b9d045-0f29-42c6-b394-53a075824ec8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6pjzl\" (UID: \"13b9d045-0f29-42c6-b394-53a075824ec8\") " pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:55:04.363032 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:04.363012 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13b9d045-0f29-42c6-b394-53a075824ec8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pjzl\" (UID: \"13b9d045-0f29-42c6-b394-53a075824ec8\") " pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:55:04.517422 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:04.517396 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6pjzl" Sep 09 20:55:04.562247 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:04.562219 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:04.754751 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:04.754722 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:04.790082 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:04.789953 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6pjzl"] Sep 09 20:55:04.792530 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:55:04.792495 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b9d045_0f29_42c6_b394_53a075824ec8.slice/crio-2c9c4207b9fbd99925f2636252707cfd0f114a66ef65be8e1a3849c80a5574ec WatchSource:0}: Error finding container 2c9c4207b9fbd99925f2636252707cfd0f114a66ef65be8e1a3849c80a5574ec: Status 404 returned error can't find the container with id 2c9c4207b9fbd99925f2636252707cfd0f114a66ef65be8e1a3849c80a5574ec Sep 09 20:55:04.954492 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:04.954464 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6pjzl" event={"ID":"13b9d045-0f29-42c6-b394-53a075824ec8","Type":"ContainerStarted","Data":"c957e416d48d8d731ca7a42387d7e947b7ed31de21ec4c68e971cf854e97f2ff"} Sep 09 20:55:04.954610 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:04.954513 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6pjzl" event={"ID":"13b9d045-0f29-42c6-b394-53a075824ec8","Type":"ContainerStarted","Data":"2c9c4207b9fbd99925f2636252707cfd0f114a66ef65be8e1a3849c80a5574ec"} Sep 09 20:55:05.859232 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:05.859207 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:05.958706 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:05.958675 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6pjzl" event={"ID":"13b9d045-0f29-42c6-b394-53a075824ec8","Type":"ContainerStarted","Data":"d0a7112637e3d8181c987d25d13ee6b0010b7e08e69e99e80c89dc30acc25cb5"} Sep 09 20:55:05.972940 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:05.972919 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls\") pod \"dns-default-gb8k5\" (UID: \"81bd5151-ce78-4a7c-9ebf-b787d915430c\") " pod="openshift-dns/dns-default-gb8k5" Sep 09 20:55:05.975181 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:05.975166 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81bd5151-ce78-4a7c-9ebf-b787d915430c-metrics-tls\") pod \"dns-default-gb8k5\" (UID: \"81bd5151-ce78-4a7c-9ebf-b787d915430c\") " pod="openshift-dns/dns-default-gb8k5" Sep 09 20:55:06.250560 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:06.250482 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wtvxz\"" Sep 09 20:55:06.258480 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:06.258458 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gb8k5" Sep 09 20:55:06.537276 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:06.537203 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gb8k5"] Sep 09 20:55:06.541080 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:55:06.541051 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81bd5151_ce78_4a7c_9ebf_b787d915430c.slice/crio-5e14a5a01597e4ea4bbe6dda00646d53f3c704713a07414355950a6d8901d715 WatchSource:0}: Error finding container 5e14a5a01597e4ea4bbe6dda00646d53f3c704713a07414355950a6d8901d715: Status 404 returned error can't find the container with id 5e14a5a01597e4ea4bbe6dda00646d53f3c704713a07414355950a6d8901d715 Sep 09 20:55:06.963360 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:06.963314 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gb8k5" event={"ID":"81bd5151-ce78-4a7c-9ebf-b787d915430c","Type":"ContainerStarted","Data":"5e14a5a01597e4ea4bbe6dda00646d53f3c704713a07414355950a6d8901d715"} Sep 09 20:55:07.970552 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:07.970480 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6pjzl" event={"ID":"13b9d045-0f29-42c6-b394-53a075824ec8","Type":"ContainerStarted","Data":"ccd3e89b4850ecc06f5af45be04425c18b9213c5ac7fd83f576b68771ff9a181"} Sep 09 20:55:07.987342 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:07.987285 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6pjzl" podStartSLOduration=17.818976125 podStartE2EDuration="19.987266717s" totalImagesPullingTime="2.168290592s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:54:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:07.986931503 +0000 UTC m=+285.447573424" watchObservedRunningTime="2026-09-09 20:55:07.987266717 +0000 UTC m=+285.447908638" Sep 09 20:55:08.339745 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:08.339708 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:08.440062 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:08.440032 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:08.501639 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:08.501558 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:08.631142 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:08.631111 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:08.805307 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:08.805222 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:08.867038 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:08.867004 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:08.975645 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:08.975613 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gb8k5" event={"ID":"81bd5151-ce78-4a7c-9ebf-b787d915430c","Type":"ContainerStarted","Data":"d35e3c6c088d9eacd7527b12500b9400d363c15d0845e77c9002f02ca1ee5265"} Sep 09 20:55:08.976077 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:08.975666 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gb8k5" event={"ID":"81bd5151-ce78-4a7c-9ebf-b787d915430c","Type":"ContainerStarted","Data":"ae83473f9f42a564f4a5498642652773afe52f2900bc20fa0122f567c165a9a6"} Sep 09 20:55:08.976077 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:08.975871 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gb8k5" Sep 09 20:55:08.996960 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:08.996907 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gb8k5" podStartSLOduration=252.355096909 podStartE2EDuration="4m13.996869838s" totalImagesPullingTime="1.641772929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:08.996142579 +0000 UTC m=+286.456784501" watchObservedRunningTime="2026-09-09 20:55:08.996869838 +0000 UTC m=+286.457511760" Sep 09 20:55:09.003318 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:09.003290 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:09.060668 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:09.060599 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:09.238454 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:09.238416 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:09.298195 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:09.298171 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:09.419619 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:09.419533 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:10.015727 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:10.015696 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:10.190160 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:10.190129 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:10.285000 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:10.284920 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:11.031348 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:11.031311 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:11.675474 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:11.675438 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:11.739733 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:11.739706 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:13.038781 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:55:13.038731 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-9dfd9" podUID="f8e8b1d2-6523-403e-bace-b96b38d8449b" Sep 09 20:55:14.749129 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:14.749100 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:15.039385 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:15.039298 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:17.259202 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:17.259166 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:18.233988 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:18.233953 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:18.334759 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:18.334728 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:18.433401 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:18.433362 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:18.534838 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:18.534739 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:18.981742 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:18.981712 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gb8k5" Sep 09 20:55:19.012268 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:19.012238 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:19.441334 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:19.441301 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:21.521626 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:21.521596 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:21.622274 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:21.622244 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:21.865108 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:21.865003 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:22.945463 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:22.945441 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tctqv_ddb3d74a-a1bb-4bda-8d0d-89320e50f357/ovn-acl-logging/0.log" Sep 09 20:55:22.945857 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:22.945493 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tctqv_ddb3d74a-a1bb-4bda-8d0d-89320e50f357/ovn-acl-logging/0.log" Sep 09 20:55:22.951708 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:22.951687 2580 kubelet.go:1648] "Image garbage collection succeeded" Sep 09 20:55:23.143388 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.143280 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55f747b7d5-t6zgp"] Sep 09 20:55:23.147440 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.146451 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:55:23.149336 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.149315 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 09 20:55:23.149336 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.149329 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 09 20:55:23.149336 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.149364 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 09 20:55:23.149336 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.149367 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4fgcx\"" Sep 09 20:55:23.149657 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.149315 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 09 20:55:23.149657 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.149315 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 09 20:55:23.149657 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.149407 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 09 20:55:23.149657 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.149556 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 09 20:55:23.160718 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.160702 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55f747b7d5-t6zgp"] Sep 09 20:55:23.181189 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.181163 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:23.198062 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.198024 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn9cw\" (UniqueName: \"kubernetes.io/projected/0d039639-63cc-4d84-816c-b36490f0f1f7-kube-api-access-gn9cw\") pod \"console-55f747b7d5-t6zgp\" (UID: \"0d039639-63cc-4d84-816c-b36490f0f1f7\") " pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:55:23.198062 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.198049 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d039639-63cc-4d84-816c-b36490f0f1f7-oauth-serving-cert\") pod \"console-55f747b7d5-t6zgp\" (UID: \"0d039639-63cc-4d84-816c-b36490f0f1f7\") " pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:55:23.198149 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.198079 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d039639-63cc-4d84-816c-b36490f0f1f7-console-serving-cert\") pod \"console-55f747b7d5-t6zgp\" (UID: \"0d039639-63cc-4d84-816c-b36490f0f1f7\") " pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:55:23.198149 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.198098 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d039639-63cc-4d84-816c-b36490f0f1f7-console-config\") pod \"console-55f747b7d5-t6zgp\" (UID: \"0d039639-63cc-4d84-816c-b36490f0f1f7\") " pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:55:23.198212 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.198151 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d039639-63cc-4d84-816c-b36490f0f1f7-console-oauth-config\") pod \"console-55f747b7d5-t6zgp\" (UID: \"0d039639-63cc-4d84-816c-b36490f0f1f7\") " pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:55:23.198212 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.198165 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d039639-63cc-4d84-816c-b36490f0f1f7-service-ca\") pod \"console-55f747b7d5-t6zgp\" (UID: \"0d039639-63cc-4d84-816c-b36490f0f1f7\") " pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:55:23.281505 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.281485 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:23.298348 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.298332 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d039639-63cc-4d84-816c-b36490f0f1f7-console-oauth-config\") pod \"console-55f747b7d5-t6zgp\" (UID: \"0d039639-63cc-4d84-816c-b36490f0f1f7\") " pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:55:23.298425 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.298354 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d039639-63cc-4d84-816c-b36490f0f1f7-service-ca\") pod \"console-55f747b7d5-t6zgp\" (UID: \"0d039639-63cc-4d84-816c-b36490f0f1f7\") " pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:55:23.298425 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.298380 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn9cw\" (UniqueName: \"kubernetes.io/projected/0d039639-63cc-4d84-816c-b36490f0f1f7-kube-api-access-gn9cw\") pod \"console-55f747b7d5-t6zgp\" (UID: \"0d039639-63cc-4d84-816c-b36490f0f1f7\") " pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:55:23.298425 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.298400 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d039639-63cc-4d84-816c-b36490f0f1f7-oauth-serving-cert\") pod \"console-55f747b7d5-t6zgp\" (UID: \"0d039639-63cc-4d84-816c-b36490f0f1f7\") " pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:55:23.298555 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.298426 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d039639-63cc-4d84-816c-b36490f0f1f7-console-serving-cert\") pod \"console-55f747b7d5-t6zgp\" (UID: \"0d039639-63cc-4d84-816c-b36490f0f1f7\") " pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:55:23.298555 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.298444 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d039639-63cc-4d84-816c-b36490f0f1f7-console-config\") pod \"console-55f747b7d5-t6zgp\" (UID: \"0d039639-63cc-4d84-816c-b36490f0f1f7\") " pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:55:23.299032 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.299011 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d039639-63cc-4d84-816c-b36490f0f1f7-service-ca\") pod \"console-55f747b7d5-t6zgp\" (UID: \"0d039639-63cc-4d84-816c-b36490f0f1f7\") " pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:55:23.299128 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.299055 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d039639-63cc-4d84-816c-b36490f0f1f7-oauth-serving-cert\") pod \"console-55f747b7d5-t6zgp\" (UID: \"0d039639-63cc-4d84-816c-b36490f0f1f7\") " pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:55:23.299274 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.299136 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d039639-63cc-4d84-816c-b36490f0f1f7-console-config\") pod \"console-55f747b7d5-t6zgp\" (UID: \"0d039639-63cc-4d84-816c-b36490f0f1f7\") " pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:55:23.300610 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.300580 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d039639-63cc-4d84-816c-b36490f0f1f7-console-oauth-config\") pod \"console-55f747b7d5-t6zgp\" (UID: \"0d039639-63cc-4d84-816c-b36490f0f1f7\") " pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:55:23.300793 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.300775 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d039639-63cc-4d84-816c-b36490f0f1f7-console-serving-cert\") pod \"console-55f747b7d5-t6zgp\" (UID: \"0d039639-63cc-4d84-816c-b36490f0f1f7\") " pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:55:23.307082 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.307060 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn9cw\" (UniqueName: \"kubernetes.io/projected/0d039639-63cc-4d84-816c-b36490f0f1f7-kube-api-access-gn9cw\") pod \"console-55f747b7d5-t6zgp\" (UID: \"0d039639-63cc-4d84-816c-b36490f0f1f7\") " pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:55:23.454882 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.454834 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:55:23.720152 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:23.720082 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55f747b7d5-t6zgp"] Sep 09 20:55:23.722680 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:55:23.722655 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d039639_63cc_4d84_816c_b36490f0f1f7.slice/crio-47638eab75c1d37b8073ac2e232bab83cac320f8ef06a3889d183334354d0698 WatchSource:0}: Error finding container 47638eab75c1d37b8073ac2e232bab83cac320f8ef06a3889d183334354d0698: Status 404 returned error can't find the container with id 47638eab75c1d37b8073ac2e232bab83cac320f8ef06a3889d183334354d0698 Sep 09 20:55:24.021399 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:24.021349 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f747b7d5-t6zgp" event={"ID":"0d039639-63cc-4d84-816c-b36490f0f1f7","Type":"ContainerStarted","Data":"47638eab75c1d37b8073ac2e232bab83cac320f8ef06a3889d183334354d0698"} Sep 09 20:55:24.111325 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:24.111306 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:25.037809 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:25.037785 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:55:25.164979 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:25.164950 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:25.272045 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:25.272018 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:25.329858 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:25.329783 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:25.658799 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:25.658717 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:25.880951 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:25.880918 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:25.930477 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:25.930407 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hzjk9"] Sep 09 20:55:25.933476 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:25.933458 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:25.936296 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:25.936150 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 09 20:55:25.936296 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:25.936185 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 09 20:55:25.936296 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:25.936150 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 09 20:55:25.936837 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:25.936813 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-95jk5\"" Sep 09 20:55:25.937172 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:25.937139 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 09 20:55:25.937289 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:25.937139 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 09 20:55:25.937289 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:25.937139 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 09 20:55:25.962266 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:25.962242 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:26.016000 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.015975 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/57beebb6-a10e-45af-92c9-2d1a066a381e-root\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.016100 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.016018 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57beebb6-a10e-45af-92c9-2d1a066a381e-metrics-client-ca\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.016100 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.016045 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57beebb6-a10e-45af-92c9-2d1a066a381e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.016194 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.016095 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/57beebb6-a10e-45af-92c9-2d1a066a381e-node-exporter-accelerators-collector-config\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.016285 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.016265 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/57beebb6-a10e-45af-92c9-2d1a066a381e-node-exporter-tls\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.016344 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.016299 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wddj\" (UniqueName: \"kubernetes.io/projected/57beebb6-a10e-45af-92c9-2d1a066a381e-kube-api-access-7wddj\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.016400 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.016357 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/57beebb6-a10e-45af-92c9-2d1a066a381e-node-exporter-textfile\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.016400 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.016390 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57beebb6-a10e-45af-92c9-2d1a066a381e-sys\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.016494 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.016479 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/57beebb6-a10e-45af-92c9-2d1a066a381e-node-exporter-wtmp\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.026967 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.026939 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:26.117410 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.117384 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/57beebb6-a10e-45af-92c9-2d1a066a381e-node-exporter-wtmp\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.117823 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.117416 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/57beebb6-a10e-45af-92c9-2d1a066a381e-root\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.117823 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.117450 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57beebb6-a10e-45af-92c9-2d1a066a381e-metrics-client-ca\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.117823 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.117475 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57beebb6-a10e-45af-92c9-2d1a066a381e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.117823 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.117502 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/57beebb6-a10e-45af-92c9-2d1a066a381e-node-exporter-accelerators-collector-config\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.117823 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.117509 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/57beebb6-a10e-45af-92c9-2d1a066a381e-root\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.117823 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.117531 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/57beebb6-a10e-45af-92c9-2d1a066a381e-node-exporter-tls\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.117823 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.117556 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wddj\" (UniqueName: \"kubernetes.io/projected/57beebb6-a10e-45af-92c9-2d1a066a381e-kube-api-access-7wddj\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.117823 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.117574 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/57beebb6-a10e-45af-92c9-2d1a066a381e-node-exporter-wtmp\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.117823 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.117592 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/57beebb6-a10e-45af-92c9-2d1a066a381e-node-exporter-textfile\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.117823 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.117626 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57beebb6-a10e-45af-92c9-2d1a066a381e-sys\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.117823 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:55:26.117647 2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 09 20:55:26.117823 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:55:26.117709 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57beebb6-a10e-45af-92c9-2d1a066a381e-node-exporter-tls podName:57beebb6-a10e-45af-92c9-2d1a066a381e nodeName:}" failed. No retries permitted until 2026-09-09 20:55:26.617690422 +0000 UTC m=+304.078332328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/57beebb6-a10e-45af-92c9-2d1a066a381e-node-exporter-tls") pod "node-exporter-hzjk9" (UID: "57beebb6-a10e-45af-92c9-2d1a066a381e") : secret "node-exporter-tls" not found Sep 09 20:55:26.117823 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.117716 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57beebb6-a10e-45af-92c9-2d1a066a381e-sys\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.118443 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.118166 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/57beebb6-a10e-45af-92c9-2d1a066a381e-node-exporter-textfile\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.118680 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.118648 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/57beebb6-a10e-45af-92c9-2d1a066a381e-node-exporter-accelerators-collector-config\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.118811 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.118660 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57beebb6-a10e-45af-92c9-2d1a066a381e-metrics-client-ca\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.120866 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.120805 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57beebb6-a10e-45af-92c9-2d1a066a381e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.126569 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.126526 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wddj\" (UniqueName: \"kubernetes.io/projected/57beebb6-a10e-45af-92c9-2d1a066a381e-kube-api-access-7wddj\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.160701 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.160677 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:26.259834 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.259804 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:26.358190 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.358161 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:26.458778 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.458726 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:26.582804 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.582716 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:26.622628 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.622601 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/57beebb6-a10e-45af-92c9-2d1a066a381e-node-exporter-tls\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.625181 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.625161 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/57beebb6-a10e-45af-92c9-2d1a066a381e-node-exporter-tls\") pod \"node-exporter-hzjk9\" (UID: \"57beebb6-a10e-45af-92c9-2d1a066a381e\") " pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.843945 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.843852 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hzjk9" Sep 09 20:55:26.848278 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.848249 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:26.948339 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:26.948301 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:27.029429 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:27.029390 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:27.249789 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:55:27.249755 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57beebb6_a10e_45af_92c9_2d1a066a381e.slice/crio-4d8ae994969fd86f2a0251f3932ca4efd3e38902005cb8a459a428aa439b8af1 WatchSource:0}: Error finding container 4d8ae994969fd86f2a0251f3932ca4efd3e38902005cb8a459a428aa439b8af1: Status 404 returned error can't find the container with id 4d8ae994969fd86f2a0251f3932ca4efd3e38902005cb8a459a428aa439b8af1 Sep 09 20:55:27.254072 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:27.254056 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 20:55:27.577427 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:27.577354 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:28.037953 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:28.037884 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f747b7d5-t6zgp" event={"ID":"0d039639-63cc-4d84-816c-b36490f0f1f7","Type":"ContainerStarted","Data":"857d0243fe2a78e9917e38aa68223e257d3d9c6bf9cb1e511e490a48673f7a42"} Sep 09 20:55:28.039343 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:28.039307 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hzjk9" event={"ID":"57beebb6-a10e-45af-92c9-2d1a066a381e","Type":"ContainerStarted","Data":"4d8ae994969fd86f2a0251f3932ca4efd3e38902005cb8a459a428aa439b8af1"} Sep 09 20:55:28.046723 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:28.046693 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:28.059049 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:28.059001 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55f747b7d5-t6zgp" podStartSLOduration=1.4897931039999999 podStartE2EDuration="5.058982458s" totalImagesPullingTime="3.569189354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:28.058399808 +0000 UTC m=+305.519041733" watchObservedRunningTime="2026-09-09 20:55:28.058982458 +0000 UTC m=+305.519624380" Sep 09 20:55:28.148302 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:28.148280 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:29.043722 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:29.043692 2580 generic.go:356] "Generic (PLEG): container finished" podID="57beebb6-a10e-45af-92c9-2d1a066a381e" containerID="c449607c7a11fd445517b5258fc5730b18f1356b90036657c2848e1ae6f360ce" exitCode=0 Sep 09 20:55:29.044147 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:29.043779 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hzjk9" event={"ID":"57beebb6-a10e-45af-92c9-2d1a066a381e","Type":"ContainerDied","Data":"c449607c7a11fd445517b5258fc5730b18f1356b90036657c2848e1ae6f360ce"} Sep 09 20:55:29.972275 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:29.972236 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:30.049042 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:30.049012 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hzjk9" event={"ID":"57beebb6-a10e-45af-92c9-2d1a066a381e","Type":"ContainerStarted","Data":"2d724d6dfb41d1bd60713310deddf96c06ab29c8ea6818d0d4245c47a4278f50"} Sep 09 20:55:30.049420 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:30.049057 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hzjk9" event={"ID":"57beebb6-a10e-45af-92c9-2d1a066a381e","Type":"ContainerStarted","Data":"a0b5d62b4f7fb63e662486d5cb2db441951616eed0308c571c7bd4f68922fe36"} Sep 09 20:55:30.068715 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:30.068673 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hzjk9" podStartSLOduration=4.09348284 podStartE2EDuration="5.068658966s" totalImagesPullingTime="904.918486ms" totalInitContainerRuntime="70.25764ms" isStatefulPod=true podCreationTimestamp="2026-09-09 20:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:30.067437534 +0000 UTC m=+307.528079459" watchObservedRunningTime="2026-09-09 20:55:30.068658966 +0000 UTC m=+307.529300887" Sep 09 20:55:30.070988 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:30.070957 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:30.196850 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:30.196824 2580 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" podUID="05c7f884-449b-4624-8a43-4e464d2aa55e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 09 20:55:30.734222 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:30.734192 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:30.830058 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:30.830033 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:30.968031 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:30.968009 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:31.066581 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.066529 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54684f4fb4-vlcsw"] Sep 09 20:55:31.069569 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.069555 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:31.077989 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.077966 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 09 20:55:31.085808 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.085778 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54684f4fb4-vlcsw"] Sep 09 20:55:31.157531 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.157505 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x4kh\" (UniqueName: \"kubernetes.io/projected/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-kube-api-access-6x4kh\") pod \"console-54684f4fb4-vlcsw\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:31.157649 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.157588 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-oauth-serving-cert\") pod \"console-54684f4fb4-vlcsw\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:31.157713 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.157671 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-console-serving-cert\") pod \"console-54684f4fb4-vlcsw\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:31.157772 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.157713 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-console-oauth-config\") pod \"console-54684f4fb4-vlcsw\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:31.157772 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.157751 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-service-ca\") pod \"console-54684f4fb4-vlcsw\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:31.157855 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.157797 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-console-config\") pod \"console-54684f4fb4-vlcsw\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:31.157855 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.157840 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-trusted-ca-bundle\") pod \"console-54684f4fb4-vlcsw\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:31.258953 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.258929 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-console-serving-cert\") pod \"console-54684f4fb4-vlcsw\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:31.259057 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.258961 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-console-oauth-config\") pod \"console-54684f4fb4-vlcsw\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:31.259057 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.258976 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-service-ca\") pod \"console-54684f4fb4-vlcsw\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:31.259057 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.259012 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-console-config\") pod \"console-54684f4fb4-vlcsw\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:31.259188 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.259145 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-trusted-ca-bundle\") pod \"console-54684f4fb4-vlcsw\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:31.259188 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.259182 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x4kh\" (UniqueName: \"kubernetes.io/projected/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-kube-api-access-6x4kh\") pod \"console-54684f4fb4-vlcsw\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:31.259282 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.259219 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-oauth-serving-cert\") pod \"console-54684f4fb4-vlcsw\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:31.259646 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.259603 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-service-ca\") pod \"console-54684f4fb4-vlcsw\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:31.259873 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.259854 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-console-config\") pod \"console-54684f4fb4-vlcsw\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:31.259943 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.259911 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-oauth-serving-cert\") pod \"console-54684f4fb4-vlcsw\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:31.259981 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.259955 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-trusted-ca-bundle\") pod \"console-54684f4fb4-vlcsw\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:31.261477 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.261457 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-console-serving-cert\") pod \"console-54684f4fb4-vlcsw\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:31.261539 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.261483 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-console-oauth-config\") pod \"console-54684f4fb4-vlcsw\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:31.269899 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.269873 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x4kh\" (UniqueName: \"kubernetes.io/projected/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-kube-api-access-6x4kh\") pod \"console-54684f4fb4-vlcsw\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:31.378746 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.378697 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:31.648848 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.648788 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54684f4fb4-vlcsw"] Sep 09 20:55:31.651489 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:55:31.651468 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ef8c0e_3bfe_4b8a_845e_d432e173d3a8.slice/crio-06bb57ee0a10a16f5e322e35e59d6797f0677baa1169f46894a9ea7b934b9854 WatchSource:0}: Error finding container 06bb57ee0a10a16f5e322e35e59d6797f0677baa1169f46894a9ea7b934b9854: Status 404 returned error can't find the container with id 06bb57ee0a10a16f5e322e35e59d6797f0677baa1169f46894a9ea7b934b9854 Sep 09 20:55:31.679511 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.679486 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:31.747951 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:31.747919 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:32.059738 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.059710 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54684f4fb4-vlcsw" event={"ID":"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8","Type":"ContainerStarted","Data":"637842429c53e46136d60f721b92fcd4657c69d438e05747b0641dbfa3ce3f7f"} Sep 09 20:55:32.059925 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.059776 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54684f4fb4-vlcsw" event={"ID":"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8","Type":"ContainerStarted","Data":"06bb57ee0a10a16f5e322e35e59d6797f0677baa1169f46894a9ea7b934b9854"} Sep 09 20:55:32.080109 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.080051 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54684f4fb4-vlcsw" podStartSLOduration=1.080037173 podStartE2EDuration="1.080037173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:32.079223234 +0000 UTC m=+309.539865148" watchObservedRunningTime="2026-09-09 20:55:32.080037173 +0000 UTC m=+309.540679094" Sep 09 20:55:32.370622 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.370565 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 20:55:32.374091 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.374072 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.379419 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.379400 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 09 20:55:32.380041 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.380022 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 09 20:55:32.380096 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.380030 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 09 20:55:32.380670 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.380649 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 09 20:55:32.380909 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.380865 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 09 20:55:32.380909 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.380878 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 09 20:55:32.381055 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.381019 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dmp2qp896hj50\"" Sep 09 20:55:32.381187 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.381162 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 09 20:55:32.381249 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.381212 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 09 20:55:32.381828 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.381814 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 09 20:55:32.381884 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.381860 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 09 20:55:32.382026 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.382010 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-28ndd\"" Sep 09 20:55:32.384144 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.384123 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 09 20:55:32.384460 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.384445 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 09 20:55:32.385103 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.385087 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 09 20:55:32.385183 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.385148 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 09 20:55:32.389049 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.389029 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 09 20:55:32.400212 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.400186 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:32.405199 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.405171 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 20:55:32.467295 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.467271 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.467393 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.467297 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tspmn\" (UniqueName: \"kubernetes.io/projected/bb3935b1-6307-4609-87ae-5c21a75a7530-kube-api-access-tspmn\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.467393 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.467317 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.467393 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.467331 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.467519 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.467397 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.467519 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.467454 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.467519 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.467480 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.467519 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.467503 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.467679 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.467556 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bb3935b1-6307-4609-87ae-5c21a75a7530-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.467679 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.467571 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.467679 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.467590 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-config\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.467679 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.467603 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.467679 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.467616 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bb3935b1-6307-4609-87ae-5c21a75a7530-config-out\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.467679 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.467633 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.467679 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.467659 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.467942 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.467692 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.467942 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.467717 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.467942 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.467740 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-web-config\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.467942 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.467758 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.467942 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.467774 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.569081 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.569055 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.569165 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.569087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tspmn\" (UniqueName: \"kubernetes.io/projected/bb3935b1-6307-4609-87ae-5c21a75a7530-kube-api-access-tspmn\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.569165 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.569115 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.569165 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.569138 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.569357 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.569339 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.569430 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.569366 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.569430 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.569385 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.569430 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.569412 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.569576 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.569457 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bb3935b1-6307-4609-87ae-5c21a75a7530-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.569576 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.569499 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.569576 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.569540 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-config\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.569576 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.569564 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.569759 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.569588 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bb3935b1-6307-4609-87ae-5c21a75a7530-config-out\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.569759 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.569613 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.569759 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.569638 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.569759 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.569663 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.569759 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.569690 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.569759 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.569720 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-web-config\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.569759 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.569745 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.570262 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.569770 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.570262 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.570157 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.570673 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.570651 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.572281 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.571867 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.572376 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.572344 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.572580 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.572557 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.572921 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.572870 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.573205 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.573186 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bb3935b1-6307-4609-87ae-5c21a75a7530-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.573307 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.573255 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.573427 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.573405 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.573695 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.573672 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.574323 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.574283 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.574416 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.574345 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.574868 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.574828 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.574979 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.574965 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.575102 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.575084 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-config\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.575385 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.575367 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.575467 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.575385 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.575726 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.575710 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-web-config\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.575968 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.575953 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bb3935b1-6307-4609-87ae-5c21a75a7530-config-out\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.577370 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.577351 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tspmn\" (UniqueName: \"kubernetes.io/projected/bb3935b1-6307-4609-87ae-5c21a75a7530-kube-api-access-tspmn\") pod \"prometheus-k8s-0\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.684011 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.683962 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.986300 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:32.986275 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 20:55:32.987425 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:55:32.987406 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3935b1_6307_4609_87ae_5c21a75a7530.slice/crio-89efe532b7d9e9a9056af2f0535f87f4fcb99a0b3293f26ecec7d1ac198098d2 WatchSource:0}: Error finding container 89efe532b7d9e9a9056af2f0535f87f4fcb99a0b3293f26ecec7d1ac198098d2: Status 404 returned error can't find the container with id 89efe532b7d9e9a9056af2f0535f87f4fcb99a0b3293f26ecec7d1ac198098d2 Sep 09 20:55:33.063672 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:33.063645 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb3935b1-6307-4609-87ae-5c21a75a7530","Type":"ContainerStarted","Data":"89efe532b7d9e9a9056af2f0535f87f4fcb99a0b3293f26ecec7d1ac198098d2"} Sep 09 20:55:33.253699 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:33.253631 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:33.362939 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:33.362909 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:33.454959 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:33.454931 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:55:33.455080 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:33.454990 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:55:33.460330 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:33.460315 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:55:34.068399 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:34.068364 2580 generic.go:356] "Generic (PLEG): container finished" podID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerID="cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0" exitCode=0 Sep 09 20:55:34.068486 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:34.068456 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb3935b1-6307-4609-87ae-5c21a75a7530","Type":"ContainerDied","Data":"cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0"} Sep 09 20:55:34.072363 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:34.072343 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:55:34.828384 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:34.828353 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:34.887997 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:34.887966 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:36.711148 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:36.711110 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:36.808905 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:36.808855 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:36.908417 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:36.908385 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:37.495921 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:37.495874 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:37.572694 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:37.572669 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:38.088349 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:38.088317 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb3935b1-6307-4609-87ae-5c21a75a7530","Type":"ContainerStarted","Data":"177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931"} Sep 09 20:55:38.088743 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:38.088410 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb3935b1-6307-4609-87ae-5c21a75a7530","Type":"ContainerStarted","Data":"e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818"} Sep 09 20:55:39.094639 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:39.094605 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb3935b1-6307-4609-87ae-5c21a75a7530","Type":"ContainerStarted","Data":"21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9"} Sep 09 20:55:39.094932 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:39.094666 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb3935b1-6307-4609-87ae-5c21a75a7530","Type":"ContainerStarted","Data":"c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b"} Sep 09 20:55:40.102441 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:40.101974 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb3935b1-6307-4609-87ae-5c21a75a7530","Type":"ContainerStarted","Data":"ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1"} Sep 09 20:55:40.102441 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:40.102055 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb3935b1-6307-4609-87ae-5c21a75a7530","Type":"ContainerStarted","Data":"85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d"} Sep 09 20:55:40.136646 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:40.136595 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.490664095 podStartE2EDuration="8.136580617s" totalImagesPullingTime="5.626923688s" totalInitContainerRuntime="18.992834ms" isStatefulPod=false podCreationTimestamp="2026-09-09 20:55:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:40.135360628 +0000 UTC m=+317.596002560" watchObservedRunningTime="2026-09-09 20:55:40.136580617 +0000 UTC m=+317.597222538" Sep 09 20:55:40.196906 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:40.196862 2580 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" podUID="05c7f884-449b-4624-8a43-4e464d2aa55e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 09 20:55:40.288110 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:40.288086 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:40.393622 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:40.393558 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:41.379648 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:41.379615 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:41.380112 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:41.379711 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:41.384193 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:41.384172 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:41.693704 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:41.693670 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:41.775586 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:41.775564 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:42.111604 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:42.111554 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:55:42.156181 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:42.156158 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55f747b7d5-t6zgp"] Sep 09 20:55:42.265528 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:42.265489 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:42.366974 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:42.366874 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:42.466078 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:42.466051 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:42.684538 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:42.684490 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:48.184365 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:48.184331 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:48.256080 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:48.256049 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:50.196749 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:50.196716 2580 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" podUID="05c7f884-449b-4624-8a43-4e464d2aa55e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 09 20:55:50.197206 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:50.196781 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" Sep 09 20:55:50.197359 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:50.197324 2580 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"1bad55293f78f403ab2733c9710ba1721c5128cbb6e07204b4ac49b31ee630ca"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" containerMessage="Container service-proxy failed liveness probe, will be restarted" Sep 09 20:55:50.197430 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:50.197385 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" podUID="05c7f884-449b-4624-8a43-4e464d2aa55e" containerName="service-proxy" containerID="cri-o://1bad55293f78f403ab2733c9710ba1721c5128cbb6e07204b4ac49b31ee630ca" gracePeriod=30 Sep 09 20:55:51.137908 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:51.137867 2580 generic.go:356] "Generic (PLEG): container finished" podID="05c7f884-449b-4624-8a43-4e464d2aa55e" containerID="1bad55293f78f403ab2733c9710ba1721c5128cbb6e07204b4ac49b31ee630ca" exitCode=2 Sep 09 20:55:51.138069 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:51.137942 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" event={"ID":"05c7f884-449b-4624-8a43-4e464d2aa55e","Type":"ContainerDied","Data":"1bad55293f78f403ab2733c9710ba1721c5128cbb6e07204b4ac49b31ee630ca"} Sep 09 20:55:51.138069 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:51.137989 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d6d8f9846-ncmmt" event={"ID":"05c7f884-449b-4624-8a43-4e464d2aa55e","Type":"ContainerStarted","Data":"7814a9f1ce204abf38476e833e8800037a1b4aad46942e3bc0b76bbc9973abc3"} Sep 09 20:55:57.166185 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:57.166152 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:57.263510 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:55:57.263480 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:56:07.175715 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:07.175678 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55f747b7d5-t6zgp" podUID="0d039639-63cc-4d84-816c-b36490f0f1f7" containerName="console" containerID="cri-o://857d0243fe2a78e9917e38aa68223e257d3d9c6bf9cb1e511e490a48673f7a42" gracePeriod=15 Sep 09 20:56:07.411988 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:07.411967 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:56:07.519203 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:07.519176 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d039639-63cc-4d84-816c-b36490f0f1f7-console-oauth-config\") pod \"0d039639-63cc-4d84-816c-b36490f0f1f7\" (UID: \"0d039639-63cc-4d84-816c-b36490f0f1f7\") " Sep 09 20:56:07.519318 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:07.519238 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d039639-63cc-4d84-816c-b36490f0f1f7-service-ca\") pod \"0d039639-63cc-4d84-816c-b36490f0f1f7\" (UID: \"0d039639-63cc-4d84-816c-b36490f0f1f7\") " Sep 09 20:56:07.519318 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:07.519264 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d039639-63cc-4d84-816c-b36490f0f1f7-console-config\") pod \"0d039639-63cc-4d84-816c-b36490f0f1f7\" (UID: \"0d039639-63cc-4d84-816c-b36490f0f1f7\") " Sep 09 20:56:07.519318 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:07.519281 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d039639-63cc-4d84-816c-b36490f0f1f7-console-serving-cert\") pod \"0d039639-63cc-4d84-816c-b36490f0f1f7\" (UID: \"0d039639-63cc-4d84-816c-b36490f0f1f7\") " Sep 09 20:56:07.519318 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:07.519303 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gn9cw\" (UniqueName: \"kubernetes.io/projected/0d039639-63cc-4d84-816c-b36490f0f1f7-kube-api-access-gn9cw\") pod \"0d039639-63cc-4d84-816c-b36490f0f1f7\" (UID: \"0d039639-63cc-4d84-816c-b36490f0f1f7\") " Sep 09 20:56:07.519513 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:07.519344 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d039639-63cc-4d84-816c-b36490f0f1f7-oauth-serving-cert\") pod \"0d039639-63cc-4d84-816c-b36490f0f1f7\" (UID: \"0d039639-63cc-4d84-816c-b36490f0f1f7\") " Sep 09 20:56:07.519619 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:07.519588 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d039639-63cc-4d84-816c-b36490f0f1f7-service-ca" (OuterVolumeSpecName: "service-ca") pod "0d039639-63cc-4d84-816c-b36490f0f1f7" (UID: "0d039639-63cc-4d84-816c-b36490f0f1f7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:56:07.519696 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:07.519611 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d039639-63cc-4d84-816c-b36490f0f1f7-console-config" (OuterVolumeSpecName: "console-config") pod "0d039639-63cc-4d84-816c-b36490f0f1f7" (UID: "0d039639-63cc-4d84-816c-b36490f0f1f7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:56:07.519854 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:07.519825 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d039639-63cc-4d84-816c-b36490f0f1f7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0d039639-63cc-4d84-816c-b36490f0f1f7" (UID: "0d039639-63cc-4d84-816c-b36490f0f1f7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:56:07.521566 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:07.521535 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d039639-63cc-4d84-816c-b36490f0f1f7-kube-api-access-gn9cw" (OuterVolumeSpecName: "kube-api-access-gn9cw") pod "0d039639-63cc-4d84-816c-b36490f0f1f7" (UID: "0d039639-63cc-4d84-816c-b36490f0f1f7"). InnerVolumeSpecName "kube-api-access-gn9cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:56:07.521566 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:07.521551 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d039639-63cc-4d84-816c-b36490f0f1f7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0d039639-63cc-4d84-816c-b36490f0f1f7" (UID: "0d039639-63cc-4d84-816c-b36490f0f1f7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:56:07.521699 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:07.521568 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d039639-63cc-4d84-816c-b36490f0f1f7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0d039639-63cc-4d84-816c-b36490f0f1f7" (UID: "0d039639-63cc-4d84-816c-b36490f0f1f7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:56:07.620605 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:07.620578 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d039639-63cc-4d84-816c-b36490f0f1f7-console-oauth-config\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:07.620605 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:07.620602 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d039639-63cc-4d84-816c-b36490f0f1f7-service-ca\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:07.620732 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:07.620616 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d039639-63cc-4d84-816c-b36490f0f1f7-console-config\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:07.620732 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:07.620627 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d039639-63cc-4d84-816c-b36490f0f1f7-console-serving-cert\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:07.620732 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:07.620638 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn9cw\" (UniqueName: \"kubernetes.io/projected/0d039639-63cc-4d84-816c-b36490f0f1f7-kube-api-access-gn9cw\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:07.620732 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:07.620649 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d039639-63cc-4d84-816c-b36490f0f1f7-oauth-serving-cert\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:08.191917 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:08.191868 2580 generic.go:356] "Generic (PLEG): container finished" podID="0d039639-63cc-4d84-816c-b36490f0f1f7" containerID="857d0243fe2a78e9917e38aa68223e257d3d9c6bf9cb1e511e490a48673f7a42" exitCode=0 Sep 09 20:56:08.192364 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:08.191940 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f747b7d5-t6zgp" Sep 09 20:56:08.192364 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:08.191933 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f747b7d5-t6zgp" event={"ID":"0d039639-63cc-4d84-816c-b36490f0f1f7","Type":"ContainerDied","Data":"857d0243fe2a78e9917e38aa68223e257d3d9c6bf9cb1e511e490a48673f7a42"} Sep 09 20:56:08.192364 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:08.192053 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f747b7d5-t6zgp" event={"ID":"0d039639-63cc-4d84-816c-b36490f0f1f7","Type":"ContainerDied","Data":"47638eab75c1d37b8073ac2e232bab83cac320f8ef06a3889d183334354d0698"} Sep 09 20:56:08.192364 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:08.192075 2580 scope.go:117] "RemoveContainer" containerID="857d0243fe2a78e9917e38aa68223e257d3d9c6bf9cb1e511e490a48673f7a42" Sep 09 20:56:08.200668 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:08.200645 2580 scope.go:117] "RemoveContainer" containerID="857d0243fe2a78e9917e38aa68223e257d3d9c6bf9cb1e511e490a48673f7a42" Sep 09 20:56:08.200964 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:56:08.200946 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857d0243fe2a78e9917e38aa68223e257d3d9c6bf9cb1e511e490a48673f7a42\": container with ID starting with 857d0243fe2a78e9917e38aa68223e257d3d9c6bf9cb1e511e490a48673f7a42 not found: ID does not exist" containerID="857d0243fe2a78e9917e38aa68223e257d3d9c6bf9cb1e511e490a48673f7a42" Sep 09 20:56:08.201020 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:08.200972 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857d0243fe2a78e9917e38aa68223e257d3d9c6bf9cb1e511e490a48673f7a42"} err="failed to get container status \"857d0243fe2a78e9917e38aa68223e257d3d9c6bf9cb1e511e490a48673f7a42\": rpc error: code = NotFound desc = could not find container \"857d0243fe2a78e9917e38aa68223e257d3d9c6bf9cb1e511e490a48673f7a42\": container with ID starting with 857d0243fe2a78e9917e38aa68223e257d3d9c6bf9cb1e511e490a48673f7a42 not found: ID does not exist" Sep 09 20:56:08.213060 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:08.213042 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55f747b7d5-t6zgp"] Sep 09 20:56:08.234679 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:08.234659 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55f747b7d5-t6zgp"] Sep 09 20:56:09.042236 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:09.042212 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d039639-63cc-4d84-816c-b36490f0f1f7" path="/var/lib/kubelet/pods/0d039639-63cc-4d84-816c-b36490f0f1f7/volumes" Sep 09 20:56:11.695296 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:11.695264 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:56:11.778021 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:11.777997 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:56:32.472932 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:32.472875 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:56:32.535404 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:32.535383 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:56:32.684726 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:32.684705 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:32.704412 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:32.704394 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:33.284258 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:33.284237 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:36.015232 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:36.015199 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs\") pod \"network-metrics-daemon-9dfd9\" (UID: \"f8e8b1d2-6523-403e-bace-b96b38d8449b\") " pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:56:36.018584 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:36.018551 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8e8b1d2-6523-403e-bace-b96b38d8449b-metrics-certs\") pod \"network-metrics-daemon-9dfd9\" (UID: \"f8e8b1d2-6523-403e-bace-b96b38d8449b\") " pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:56:36.140984 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:36.140957 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kjjjq\"" Sep 09 20:56:36.148899 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:36.148871 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dfd9" Sep 09 20:56:36.427647 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:36.427579 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9dfd9"] Sep 09 20:56:36.430256 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:56:36.430226 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e8b1d2_6523_403e_bace_b96b38d8449b.slice/crio-fa84a98ef9e113e84a284034bd20c1f359b5b57ef6d196e83d671b1d9122fc15 WatchSource:0}: Error finding container fa84a98ef9e113e84a284034bd20c1f359b5b57ef6d196e83d671b1d9122fc15: Status 404 returned error can't find the container with id fa84a98ef9e113e84a284034bd20c1f359b5b57ef6d196e83d671b1d9122fc15 Sep 09 20:56:37.283568 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:37.283528 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9dfd9" event={"ID":"f8e8b1d2-6523-403e-bace-b96b38d8449b","Type":"ContainerStarted","Data":"fa84a98ef9e113e84a284034bd20c1f359b5b57ef6d196e83d671b1d9122fc15"} Sep 09 20:56:38.288628 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:38.288603 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9dfd9" event={"ID":"f8e8b1d2-6523-403e-bace-b96b38d8449b","Type":"ContainerStarted","Data":"01fa81f621af3045141fbefc612eb81fbfaa18b97a3824602a21158e8e69b86a"} Sep 09 20:56:38.288976 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:38.288647 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9dfd9" event={"ID":"f8e8b1d2-6523-403e-bace-b96b38d8449b","Type":"ContainerStarted","Data":"c68e12a3dd2422f1d0e8f2058ccacb95ea4ee9a3bcfef0add864078c41561c37"} Sep 09 20:56:38.306876 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:38.306746 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9dfd9" podStartSLOduration=374.183139616 podStartE2EDuration="6m15.306729986s" totalImagesPullingTime="1.12359037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:56:38.306169353 +0000 UTC m=+375.766811290" watchObservedRunningTime="2026-09-09 20:56:38.306729986 +0000 UTC m=+375.767371908" Sep 09 20:56:38.404229 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:38.404206 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:56:38.506303 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:38.506273 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:56:39.039530 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:39.039505 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:56:39.613210 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:39.613184 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:56:45.423552 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:45.423520 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:56:45.615756 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:45.615726 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:56:51.006227 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.006193 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 20:56:51.006639 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.006538 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="prometheus" containerID="cri-o://177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931" gracePeriod=600 Sep 09 20:56:51.006639 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.006566 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1" gracePeriod=600 Sep 09 20:56:51.006766 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.006546 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="thanos-sidecar" containerID="cri-o://c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b" gracePeriod=600 Sep 09 20:56:51.006766 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.006546 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="kube-rbac-proxy" containerID="cri-o://85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d" gracePeriod=600 Sep 09 20:56:51.006766 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.006575 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="config-reloader" containerID="cri-o://e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818" gracePeriod=600 Sep 09 20:56:51.006766 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.006574 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="kube-rbac-proxy-web" containerID="cri-o://21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9" gracePeriod=600 Sep 09 20:56:51.239752 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.239728 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.326245 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.326185 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-thanos-prometheus-http-client-file\") pod \"bb3935b1-6307-4609-87ae-5c21a75a7530\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " Sep 09 20:56:51.326245 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.326216 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-trusted-ca-bundle\") pod \"bb3935b1-6307-4609-87ae-5c21a75a7530\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " Sep 09 20:56:51.326245 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.326241 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tspmn\" (UniqueName: \"kubernetes.io/projected/bb3935b1-6307-4609-87ae-5c21a75a7530-kube-api-access-tspmn\") pod \"bb3935b1-6307-4609-87ae-5c21a75a7530\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " Sep 09 20:56:51.326464 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.326255 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bb3935b1-6307-4609-87ae-5c21a75a7530-tls-assets\") pod \"bb3935b1-6307-4609-87ae-5c21a75a7530\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " Sep 09 20:56:51.326464 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.326269 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-web-config\") pod \"bb3935b1-6307-4609-87ae-5c21a75a7530\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " Sep 09 20:56:51.326464 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.326356 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-kube-rbac-proxy\") pod \"bb3935b1-6307-4609-87ae-5c21a75a7530\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " Sep 09 20:56:51.326464 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.326396 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"bb3935b1-6307-4609-87ae-5c21a75a7530\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " Sep 09 20:56:51.326464 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.326430 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-configmap-serving-certs-ca-bundle\") pod \"bb3935b1-6307-4609-87ae-5c21a75a7530\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " Sep 09 20:56:51.326707 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.326470 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-metrics-client-certs\") pod \"bb3935b1-6307-4609-87ae-5c21a75a7530\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " Sep 09 20:56:51.326707 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.326499 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-k8s-rulefiles-2\") pod \"bb3935b1-6307-4609-87ae-5c21a75a7530\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " Sep 09 20:56:51.326707 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.326527 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-configmap-kubelet-serving-ca-bundle\") pod \"bb3935b1-6307-4609-87ae-5c21a75a7530\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " Sep 09 20:56:51.326707 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.326557 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bb3935b1-6307-4609-87ae-5c21a75a7530-config-out\") pod \"bb3935b1-6307-4609-87ae-5c21a75a7530\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " Sep 09 20:56:51.326707 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.326618 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-k8s-rulefiles-0\") pod \"bb3935b1-6307-4609-87ae-5c21a75a7530\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " Sep 09 20:56:51.326707 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.326626 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "bb3935b1-6307-4609-87ae-5c21a75a7530" (UID: "bb3935b1-6307-4609-87ae-5c21a75a7530"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:56:51.326707 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.326644 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"bb3935b1-6307-4609-87ae-5c21a75a7530\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " Sep 09 20:56:51.326707 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.326673 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-configmap-metrics-client-ca\") pod \"bb3935b1-6307-4609-87ae-5c21a75a7530\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " Sep 09 20:56:51.326707 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.326700 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-k8s-db\") pod \"bb3935b1-6307-4609-87ae-5c21a75a7530\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " Sep 09 20:56:51.327338 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.326726 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-prometheus-k8s-tls\") pod \"bb3935b1-6307-4609-87ae-5c21a75a7530\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " Sep 09 20:56:51.327338 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.326755 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-config\") pod \"bb3935b1-6307-4609-87ae-5c21a75a7530\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " Sep 09 20:56:51.327338 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.326785 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-grpc-tls\") pod \"bb3935b1-6307-4609-87ae-5c21a75a7530\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " Sep 09 20:56:51.327338 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.326819 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-k8s-rulefiles-1\") pod \"bb3935b1-6307-4609-87ae-5c21a75a7530\" (UID: \"bb3935b1-6307-4609-87ae-5c21a75a7530\") " Sep 09 20:56:51.327338 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.327007 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "bb3935b1-6307-4609-87ae-5c21a75a7530" (UID: "bb3935b1-6307-4609-87ae-5c21a75a7530"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:56:51.327338 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.327062 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:51.327338 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.327280 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "bb3935b1-6307-4609-87ae-5c21a75a7530" (UID: "bb3935b1-6307-4609-87ae-5c21a75a7530"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:56:51.327658 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.327354 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "bb3935b1-6307-4609-87ae-5c21a75a7530" (UID: "bb3935b1-6307-4609-87ae-5c21a75a7530"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:56:51.328092 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.328059 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "bb3935b1-6307-4609-87ae-5c21a75a7530" (UID: "bb3935b1-6307-4609-87ae-5c21a75a7530"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:56:51.328873 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.328812 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "bb3935b1-6307-4609-87ae-5c21a75a7530" (UID: "bb3935b1-6307-4609-87ae-5c21a75a7530"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:56:51.329012 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.328915 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "bb3935b1-6307-4609-87ae-5c21a75a7530" (UID: "bb3935b1-6307-4609-87ae-5c21a75a7530"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:56:51.329933 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.329881 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "bb3935b1-6307-4609-87ae-5c21a75a7530" (UID: "bb3935b1-6307-4609-87ae-5c21a75a7530"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:56:51.330073 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.330046 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "bb3935b1-6307-4609-87ae-5c21a75a7530" (UID: "bb3935b1-6307-4609-87ae-5c21a75a7530"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:56:51.330332 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.330299 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "bb3935b1-6307-4609-87ae-5c21a75a7530" (UID: "bb3935b1-6307-4609-87ae-5c21a75a7530"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:56:51.330393 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.330294 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3935b1-6307-4609-87ae-5c21a75a7530-config-out" (OuterVolumeSpecName: "config-out") pod "bb3935b1-6307-4609-87ae-5c21a75a7530" (UID: "bb3935b1-6307-4609-87ae-5c21a75a7530"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:56:51.330751 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.330730 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "bb3935b1-6307-4609-87ae-5c21a75a7530" (UID: "bb3935b1-6307-4609-87ae-5c21a75a7530"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:56:51.330751 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.330743 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3935b1-6307-4609-87ae-5c21a75a7530-kube-api-access-tspmn" (OuterVolumeSpecName: "kube-api-access-tspmn") pod "bb3935b1-6307-4609-87ae-5c21a75a7530" (UID: "bb3935b1-6307-4609-87ae-5c21a75a7530"). InnerVolumeSpecName "kube-api-access-tspmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:56:51.330751 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.330727 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3935b1-6307-4609-87ae-5c21a75a7530-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "bb3935b1-6307-4609-87ae-5c21a75a7530" (UID: "bb3935b1-6307-4609-87ae-5c21a75a7530"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:56:51.331259 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.331233 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "bb3935b1-6307-4609-87ae-5c21a75a7530" (UID: "bb3935b1-6307-4609-87ae-5c21a75a7530"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:56:51.331328 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.331253 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "bb3935b1-6307-4609-87ae-5c21a75a7530" (UID: "bb3935b1-6307-4609-87ae-5c21a75a7530"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:56:51.332205 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.332178 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-config" (OuterVolumeSpecName: "config") pod "bb3935b1-6307-4609-87ae-5c21a75a7530" (UID: "bb3935b1-6307-4609-87ae-5c21a75a7530"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:56:51.332292 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.332237 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "bb3935b1-6307-4609-87ae-5c21a75a7530" (UID: "bb3935b1-6307-4609-87ae-5c21a75a7530"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:56:51.332350 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.332332 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "bb3935b1-6307-4609-87ae-5c21a75a7530" (UID: "bb3935b1-6307-4609-87ae-5c21a75a7530"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:56:51.341347 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.341323 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-web-config" (OuterVolumeSpecName: "web-config") pod "bb3935b1-6307-4609-87ae-5c21a75a7530" (UID: "bb3935b1-6307-4609-87ae-5c21a75a7530"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:56:51.343408 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.343384 2580 generic.go:356] "Generic (PLEG): container finished" podID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerID="ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1" exitCode=0 Sep 09 20:56:51.343408 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.343406 2580 generic.go:356] "Generic (PLEG): container finished" podID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerID="85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d" exitCode=0 Sep 09 20:56:51.343559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.343417 2580 generic.go:356] "Generic (PLEG): container finished" podID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerID="21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9" exitCode=0 Sep 09 20:56:51.343559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.343414 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb3935b1-6307-4609-87ae-5c21a75a7530","Type":"ContainerDied","Data":"ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1"} Sep 09 20:56:51.343559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.343466 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.343559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.343425 2580 generic.go:356] "Generic (PLEG): container finished" podID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerID="c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b" exitCode=0 Sep 09 20:56:51.343559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.343520 2580 generic.go:356] "Generic (PLEG): container finished" podID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerID="e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818" exitCode=0 Sep 09 20:56:51.343559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.343508 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb3935b1-6307-4609-87ae-5c21a75a7530","Type":"ContainerDied","Data":"85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d"} Sep 09 20:56:51.343559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.343529 2580 generic.go:356] "Generic (PLEG): container finished" podID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerID="177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931" exitCode=0 Sep 09 20:56:51.343559 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.343522 2580 scope.go:117] "RemoveContainer" containerID="ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1" Sep 09 20:56:51.343942 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.343569 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb3935b1-6307-4609-87ae-5c21a75a7530","Type":"ContainerDied","Data":"21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9"} Sep 09 20:56:51.343942 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.343604 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb3935b1-6307-4609-87ae-5c21a75a7530","Type":"ContainerDied","Data":"c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b"} Sep 09 20:56:51.343942 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.343641 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb3935b1-6307-4609-87ae-5c21a75a7530","Type":"ContainerDied","Data":"e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818"} Sep 09 20:56:51.343942 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.343683 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb3935b1-6307-4609-87ae-5c21a75a7530","Type":"ContainerDied","Data":"177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931"} Sep 09 20:56:51.343942 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.343715 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb3935b1-6307-4609-87ae-5c21a75a7530","Type":"ContainerDied","Data":"89efe532b7d9e9a9056af2f0535f87f4fcb99a0b3293f26ecec7d1ac198098d2"} Sep 09 20:56:51.352824 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.352801 2580 scope.go:117] "RemoveContainer" containerID="85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d" Sep 09 20:56:51.360442 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.360429 2580 scope.go:117] "RemoveContainer" containerID="21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9" Sep 09 20:56:51.366724 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.366703 2580 scope.go:117] "RemoveContainer" containerID="c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b" Sep 09 20:56:51.372948 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.372930 2580 scope.go:117] "RemoveContainer" containerID="e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818" Sep 09 20:56:51.379696 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.379676 2580 scope.go:117] "RemoveContainer" containerID="177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931" Sep 09 20:56:51.381501 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.381481 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 20:56:51.386845 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.386824 2580 scope.go:117] "RemoveContainer" containerID="cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0" Sep 09 20:56:51.394178 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.394158 2580 scope.go:117] "RemoveContainer" containerID="ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1" Sep 09 20:56:51.394485 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:56:51.394456 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1\": container with ID starting with ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1 not found: ID does not exist" containerID="ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1" Sep 09 20:56:51.394555 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.394484 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1"} err="failed to get container status \"ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1\": rpc error: code = NotFound desc = could not find container \"ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1\": container with ID starting with ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1 not found: ID does not exist" Sep 09 20:56:51.394555 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.394503 2580 scope.go:117] "RemoveContainer" containerID="85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d" Sep 09 20:56:51.394677 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.394659 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 20:56:51.394751 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:56:51.394731 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d\": container with ID starting with 85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d not found: ID does not exist" containerID="85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d" Sep 09 20:56:51.394809 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.394757 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d"} err="failed to get container status \"85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d\": rpc error: code = NotFound desc = could not find container \"85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d\": container with ID starting with 85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d not found: ID does not exist" Sep 09 20:56:51.394809 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.394772 2580 scope.go:117] "RemoveContainer" containerID="21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9" Sep 09 20:56:51.395020 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:56:51.395006 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9\": container with ID starting with 21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9 not found: ID does not exist" containerID="21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9" Sep 09 20:56:51.395073 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.395024 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9"} err="failed to get container status \"21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9\": rpc error: code = NotFound desc = could not find container \"21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9\": container with ID starting with 21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9 not found: ID does not exist" Sep 09 20:56:51.395073 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.395036 2580 scope.go:117] "RemoveContainer" containerID="c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b" Sep 09 20:56:51.395298 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:56:51.395275 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b\": container with ID starting with c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b not found: ID does not exist" containerID="c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b" Sep 09 20:56:51.395336 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.395309 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b"} err="failed to get container status \"c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b\": rpc error: code = NotFound desc = could not find container \"c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b\": container with ID starting with c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b not found: ID does not exist" Sep 09 20:56:51.395371 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.395332 2580 scope.go:117] "RemoveContainer" containerID="e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818" Sep 09 20:56:51.395577 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:56:51.395560 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818\": container with ID starting with e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818 not found: ID does not exist" containerID="e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818" Sep 09 20:56:51.395656 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.395585 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818"} err="failed to get container status \"e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818\": rpc error: code = NotFound desc = could not find container \"e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818\": container with ID starting with e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818 not found: ID does not exist" Sep 09 20:56:51.395656 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.395604 2580 scope.go:117] "RemoveContainer" containerID="177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931" Sep 09 20:56:51.395854 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:56:51.395838 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931\": container with ID starting with 177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931 not found: ID does not exist" containerID="177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931" Sep 09 20:56:51.395976 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.395859 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931"} err="failed to get container status \"177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931\": rpc error: code = NotFound desc = could not find container \"177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931\": container with ID starting with 177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931 not found: ID does not exist" Sep 09 20:56:51.395976 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.395874 2580 scope.go:117] "RemoveContainer" containerID="cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0" Sep 09 20:56:51.396119 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:56:51.396099 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0\": container with ID starting with cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0 not found: ID does not exist" containerID="cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0" Sep 09 20:56:51.396168 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.396126 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0"} err="failed to get container status \"cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0\": rpc error: code = NotFound desc = could not find container \"cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0\": container with ID starting with cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0 not found: ID does not exist" Sep 09 20:56:51.396168 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.396145 2580 scope.go:117] "RemoveContainer" containerID="ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1" Sep 09 20:56:51.396420 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.396399 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1"} err="failed to get container status \"ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1\": rpc error: code = NotFound desc = could not find container \"ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1\": container with ID starting with ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1 not found: ID does not exist" Sep 09 20:56:51.396420 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.396419 2580 scope.go:117] "RemoveContainer" containerID="85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d" Sep 09 20:56:51.396646 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.396628 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d"} err="failed to get container status \"85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d\": rpc error: code = NotFound desc = could not find container \"85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d\": container with ID starting with 85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d not found: ID does not exist" Sep 09 20:56:51.396723 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.396647 2580 scope.go:117] "RemoveContainer" containerID="21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9" Sep 09 20:56:51.396846 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.396830 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9"} err="failed to get container status \"21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9\": rpc error: code = NotFound desc = could not find container \"21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9\": container with ID starting with 21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9 not found: ID does not exist" Sep 09 20:56:51.396936 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.396848 2580 scope.go:117] "RemoveContainer" containerID="c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b" Sep 09 20:56:51.397162 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.397145 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b"} err="failed to get container status \"c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b\": rpc error: code = NotFound desc = could not find container \"c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b\": container with ID starting with c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b not found: ID does not exist" Sep 09 20:56:51.397211 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.397163 2580 scope.go:117] "RemoveContainer" containerID="e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818" Sep 09 20:56:51.397414 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.397397 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818"} err="failed to get container status \"e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818\": rpc error: code = NotFound desc = could not find container \"e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818\": container with ID starting with e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818 not found: ID does not exist" Sep 09 20:56:51.397483 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.397416 2580 scope.go:117] "RemoveContainer" containerID="177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931" Sep 09 20:56:51.397653 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.397635 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931"} err="failed to get container status \"177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931\": rpc error: code = NotFound desc = could not find container \"177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931\": container with ID starting with 177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931 not found: ID does not exist" Sep 09 20:56:51.397698 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.397654 2580 scope.go:117] "RemoveContainer" containerID="cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0" Sep 09 20:56:51.397931 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.397913 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0"} err="failed to get container status \"cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0\": rpc error: code = NotFound desc = could not find container \"cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0\": container with ID starting with cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0 not found: ID does not exist" Sep 09 20:56:51.398010 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.397931 2580 scope.go:117] "RemoveContainer" containerID="ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1" Sep 09 20:56:51.398183 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.398163 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1"} err="failed to get container status \"ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1\": rpc error: code = NotFound desc = could not find container \"ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1\": container with ID starting with ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1 not found: ID does not exist" Sep 09 20:56:51.398239 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.398184 2580 scope.go:117] "RemoveContainer" containerID="85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d" Sep 09 20:56:51.398423 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.398405 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d"} err="failed to get container status \"85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d\": rpc error: code = NotFound desc = could not find container \"85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d\": container with ID starting with 85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d not found: ID does not exist" Sep 09 20:56:51.398465 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.398425 2580 scope.go:117] "RemoveContainer" containerID="21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9" Sep 09 20:56:51.398665 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.398647 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9"} err="failed to get container status \"21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9\": rpc error: code = NotFound desc = could not find container \"21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9\": container with ID starting with 21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9 not found: ID does not exist" Sep 09 20:56:51.398718 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.398665 2580 scope.go:117] "RemoveContainer" containerID="c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b" Sep 09 20:56:51.398920 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.398878 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b"} err="failed to get container status \"c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b\": rpc error: code = NotFound desc = could not find container \"c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b\": container with ID starting with c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b not found: ID does not exist" Sep 09 20:56:51.398998 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.398924 2580 scope.go:117] "RemoveContainer" containerID="e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818" Sep 09 20:56:51.399150 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.399130 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818"} err="failed to get container status \"e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818\": rpc error: code = NotFound desc = could not find container \"e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818\": container with ID starting with e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818 not found: ID does not exist" Sep 09 20:56:51.399193 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.399151 2580 scope.go:117] "RemoveContainer" containerID="177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931" Sep 09 20:56:51.399527 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.399504 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931"} err="failed to get container status \"177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931\": rpc error: code = NotFound desc = could not find container \"177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931\": container with ID starting with 177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931 not found: ID does not exist" Sep 09 20:56:51.399607 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.399529 2580 scope.go:117] "RemoveContainer" containerID="cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0" Sep 09 20:56:51.401489 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.401466 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0"} err="failed to get container status \"cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0\": rpc error: code = NotFound desc = could not find container \"cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0\": container with ID starting with cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0 not found: ID does not exist" Sep 09 20:56:51.401489 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.401487 2580 scope.go:117] "RemoveContainer" containerID="ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1" Sep 09 20:56:51.401946 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.401928 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1"} err="failed to get container status \"ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1\": rpc error: code = NotFound desc = could not find container \"ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1\": container with ID starting with ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1 not found: ID does not exist" Sep 09 20:56:51.401946 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.401944 2580 scope.go:117] "RemoveContainer" containerID="85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d" Sep 09 20:56:51.402218 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.402202 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d"} err="failed to get container status \"85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d\": rpc error: code = NotFound desc = could not find container \"85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d\": container with ID starting with 85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d not found: ID does not exist" Sep 09 20:56:51.402274 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.402220 2580 scope.go:117] "RemoveContainer" containerID="21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9" Sep 09 20:56:51.402491 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.402472 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9"} err="failed to get container status \"21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9\": rpc error: code = NotFound desc = could not find container \"21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9\": container with ID starting with 21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9 not found: ID does not exist" Sep 09 20:56:51.402565 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.402492 2580 scope.go:117] "RemoveContainer" containerID="c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b" Sep 09 20:56:51.402757 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.402738 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b"} err="failed to get container status \"c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b\": rpc error: code = NotFound desc = could not find container \"c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b\": container with ID starting with c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b not found: ID does not exist" Sep 09 20:56:51.402757 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.402755 2580 scope.go:117] "RemoveContainer" containerID="e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818" Sep 09 20:56:51.403034 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.403016 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818"} err="failed to get container status \"e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818\": rpc error: code = NotFound desc = could not find container \"e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818\": container with ID starting with e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818 not found: ID does not exist" Sep 09 20:56:51.403113 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.403035 2580 scope.go:117] "RemoveContainer" containerID="177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931" Sep 09 20:56:51.403269 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.403251 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931"} err="failed to get container status \"177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931\": rpc error: code = NotFound desc = could not find container \"177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931\": container with ID starting with 177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931 not found: ID does not exist" Sep 09 20:56:51.403311 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.403269 2580 scope.go:117] "RemoveContainer" containerID="cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0" Sep 09 20:56:51.403464 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.403448 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0"} err="failed to get container status \"cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0\": rpc error: code = NotFound desc = could not find container \"cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0\": container with ID starting with cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0 not found: ID does not exist" Sep 09 20:56:51.403507 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.403465 2580 scope.go:117] "RemoveContainer" containerID="ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1" Sep 09 20:56:51.403643 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.403628 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1"} err="failed to get container status \"ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1\": rpc error: code = NotFound desc = could not find container \"ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1\": container with ID starting with ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1 not found: ID does not exist" Sep 09 20:56:51.403682 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.403643 2580 scope.go:117] "RemoveContainer" containerID="85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d" Sep 09 20:56:51.403818 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.403804 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d"} err="failed to get container status \"85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d\": rpc error: code = NotFound desc = could not find container \"85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d\": container with ID starting with 85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d not found: ID does not exist" Sep 09 20:56:51.403854 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.403818 2580 scope.go:117] "RemoveContainer" containerID="21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9" Sep 09 20:56:51.404016 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.404002 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9"} err="failed to get container status \"21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9\": rpc error: code = NotFound desc = could not find container \"21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9\": container with ID starting with 21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9 not found: ID does not exist" Sep 09 20:56:51.404063 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.404017 2580 scope.go:117] "RemoveContainer" containerID="c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b" Sep 09 20:56:51.404202 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.404182 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b"} err="failed to get container status \"c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b\": rpc error: code = NotFound desc = could not find container \"c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b\": container with ID starting with c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b not found: ID does not exist" Sep 09 20:56:51.404261 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.404203 2580 scope.go:117] "RemoveContainer" containerID="e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818" Sep 09 20:56:51.404404 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.404387 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818"} err="failed to get container status \"e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818\": rpc error: code = NotFound desc = could not find container \"e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818\": container with ID starting with e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818 not found: ID does not exist" Sep 09 20:56:51.404445 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.404405 2580 scope.go:117] "RemoveContainer" containerID="177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931" Sep 09 20:56:51.404574 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.404558 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931"} err="failed to get container status \"177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931\": rpc error: code = NotFound desc = could not find container \"177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931\": container with ID starting with 177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931 not found: ID does not exist" Sep 09 20:56:51.404638 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.404575 2580 scope.go:117] "RemoveContainer" containerID="cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0" Sep 09 20:56:51.404777 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.404756 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0"} err="failed to get container status \"cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0\": rpc error: code = NotFound desc = could not find container \"cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0\": container with ID starting with cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0 not found: ID does not exist" Sep 09 20:56:51.404814 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.404778 2580 scope.go:117] "RemoveContainer" containerID="ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1" Sep 09 20:56:51.405007 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.404987 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1"} err="failed to get container status \"ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1\": rpc error: code = NotFound desc = could not find container \"ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1\": container with ID starting with ae36a8978b6afbc298e33ca3b6fbc20c3c054bfe771297d0fc6c8b97cac972e1 not found: ID does not exist" Sep 09 20:56:51.405007 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.405005 2580 scope.go:117] "RemoveContainer" containerID="85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d" Sep 09 20:56:51.405224 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.405209 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d"} err="failed to get container status \"85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d\": rpc error: code = NotFound desc = could not find container \"85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d\": container with ID starting with 85e05ae5eee7b2f9eb3c82a5f4e31961e056c3b725404f293e54e7d71d53236d not found: ID does not exist" Sep 09 20:56:51.405268 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.405225 2580 scope.go:117] "RemoveContainer" containerID="21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9" Sep 09 20:56:51.405481 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.405466 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9"} err="failed to get container status \"21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9\": rpc error: code = NotFound desc = could not find container \"21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9\": container with ID starting with 21b6b55b4528eaa58e79172af5bf865e8de4cd76d582dafd73831520ac1f89d9 not found: ID does not exist" Sep 09 20:56:51.405539 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.405481 2580 scope.go:117] "RemoveContainer" containerID="c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b" Sep 09 20:56:51.405725 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.405709 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b"} err="failed to get container status \"c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b\": rpc error: code = NotFound desc = could not find container \"c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b\": container with ID starting with c6af7a319611aaa1e00f1206fbe0707e2c18dc8d9d2b961157d3686a15d1570b not found: ID does not exist" Sep 09 20:56:51.405765 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.405726 2580 scope.go:117] "RemoveContainer" containerID="e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818" Sep 09 20:56:51.405925 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.405910 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818"} err="failed to get container status \"e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818\": rpc error: code = NotFound desc = could not find container \"e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818\": container with ID starting with e381a5759bacf437b3cca31a2dc20e3985de5a65dc939a856ad2d63f08ec1818 not found: ID does not exist" Sep 09 20:56:51.405963 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.405925 2580 scope.go:117] "RemoveContainer" containerID="177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931" Sep 09 20:56:51.406118 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.406104 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931"} err="failed to get container status \"177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931\": rpc error: code = NotFound desc = could not find container \"177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931\": container with ID starting with 177e0a1db4c967cc7e2230f5688d4ee3ed592b696a46cc48c35bfea83aaed931 not found: ID does not exist" Sep 09 20:56:51.406171 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.406118 2580 scope.go:117] "RemoveContainer" containerID="cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0" Sep 09 20:56:51.406313 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.406294 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0"} err="failed to get container status \"cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0\": rpc error: code = NotFound desc = could not find container \"cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0\": container with ID starting with cb9b4250c598b6bd954bea1dab574794a37b2220bb4500aa06e15a8377f121e0 not found: ID does not exist" Sep 09 20:56:51.427781 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.427761 2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:51.427781 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.427779 2580 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-configmap-metrics-client-ca\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:51.427863 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.427789 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-k8s-db\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:51.427863 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.427798 2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:51.427863 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.427806 2580 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-config\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:51.427863 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.427814 2580 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-grpc-tls\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:51.427863 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.427822 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:51.427863 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.427830 2580 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:51.427863 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.427839 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tspmn\" (UniqueName: \"kubernetes.io/projected/bb3935b1-6307-4609-87ae-5c21a75a7530-kube-api-access-tspmn\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:51.427863 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.427847 2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bb3935b1-6307-4609-87ae-5c21a75a7530-tls-assets\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:51.427863 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.427854 2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-web-config\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:51.427863 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.427861 2580 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-kube-rbac-proxy\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:51.428140 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.427869 2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:51.428140 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.427877 2580 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:51.428140 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.427885 2580 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bb3935b1-6307-4609-87ae-5c21a75a7530-secret-metrics-client-certs\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:51.428140 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.427914 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:51.428140 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.427921 2580 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:51.428140 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.427929 2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bb3935b1-6307-4609-87ae-5c21a75a7530-config-out\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:51.428140 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.427936 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bb3935b1-6307-4609-87ae-5c21a75a7530-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:56:51.464103 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.464079 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 20:56:51.464347 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.464332 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="config-reloader" Sep 09 20:56:51.464420 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.464350 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="config-reloader" Sep 09 20:56:51.464420 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.464361 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="kube-rbac-proxy-web" Sep 09 20:56:51.464420 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.464368 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="kube-rbac-proxy-web" Sep 09 20:56:51.464420 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.464380 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d039639-63cc-4d84-816c-b36490f0f1f7" containerName="console" Sep 09 20:56:51.464420 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.464389 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d039639-63cc-4d84-816c-b36490f0f1f7" containerName="console" Sep 09 20:56:51.464420 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.464399 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="thanos-sidecar" Sep 09 20:56:51.464420 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.464407 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="thanos-sidecar" Sep 09 20:56:51.464738 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.464430 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="init-config-reloader" Sep 09 20:56:51.464738 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.464438 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="init-config-reloader" Sep 09 20:56:51.464738 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.464449 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="prometheus" Sep 09 20:56:51.464738 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.464457 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="prometheus" Sep 09 20:56:51.464738 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.464465 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="kube-rbac-proxy-thanos" Sep 09 20:56:51.464738 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.464472 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="kube-rbac-proxy-thanos" Sep 09 20:56:51.464738 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.464483 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="kube-rbac-proxy" Sep 09 20:56:51.464738 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.464492 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="kube-rbac-proxy" Sep 09 20:56:51.464738 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.464548 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="kube-rbac-proxy-web" Sep 09 20:56:51.464738 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.464559 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d039639-63cc-4d84-816c-b36490f0f1f7" containerName="console" Sep 09 20:56:51.464738 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.464568 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="config-reloader" Sep 09 20:56:51.464738 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.464576 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="thanos-sidecar" Sep 09 20:56:51.464738 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.464584 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="prometheus" Sep 09 20:56:51.464738 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.464592 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="kube-rbac-proxy-thanos" Sep 09 20:56:51.464738 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.464601 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" containerName="kube-rbac-proxy" Sep 09 20:56:51.469348 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.469333 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.480275 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.480258 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 09 20:56:51.480371 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.480295 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 09 20:56:51.480416 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.480396 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dmp2qp896hj50\"" Sep 09 20:56:51.480611 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.480598 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-28ndd\"" Sep 09 20:56:51.483675 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.483660 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 09 20:56:51.483785 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.483771 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 09 20:56:51.483865 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.483835 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 09 20:56:51.483865 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.483842 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 09 20:56:51.483865 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.483849 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 09 20:56:51.484066 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.483864 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 09 20:56:51.484066 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.483942 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 09 20:56:51.484066 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.483980 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 09 20:56:51.484066 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.484010 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 09 20:56:51.485857 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.485841 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 09 20:56:51.494398 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.494379 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 09 20:56:51.494613 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.494596 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 09 20:56:51.495149 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.495135 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 09 20:56:51.528445 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.528427 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.528490 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.528451 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.528490 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.528467 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.528490 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.528484 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.528583 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.528507 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.528583 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.528541 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.528583 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.528570 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.528667 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.528602 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.528667 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.528619 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.528667 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.528637 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-config\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.528667 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.528650 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.528774 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.528689 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-web-config\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.528774 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.528722 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.528774 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.528749 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.528857 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.528773 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.528857 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.528794 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57xg5\" (UniqueName: \"kubernetes.io/projected/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-kube-api-access-57xg5\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.528857 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.528815 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-config-out\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.528857 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.528836 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.528857 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.528854 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.529015 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.528882 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.600911 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.600844 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 20:56:51.630104 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.630082 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57xg5\" (UniqueName: \"kubernetes.io/projected/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-kube-api-access-57xg5\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.630173 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.630108 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-config-out\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.630173 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.630129 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.630173 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.630144 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.630173 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.630162 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.630383 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.630177 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.630383 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.630199 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.630383 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.630317 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.630383 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.630361 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.630633 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.630614 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.630698 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.630665 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.630744 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.630695 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.630744 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.630730 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.630850 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.630756 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.630850 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.630774 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.630850 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.630782 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.631018 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.630819 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-config\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.631018 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.630885 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.631112 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.631032 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.631112 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.631053 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-web-config\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.631112 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.631082 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.632347 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.631660 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.632347 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.632125 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.632347 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.632152 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.632347 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.632208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.634857 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.634718 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.634857 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.634817 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.635189 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.635168 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.635247 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.635226 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-config\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.635312 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.635290 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.635360 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.635347 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-config-out\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.637925 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.635602 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.637925 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.635733 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.637925 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.635933 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-web-config\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.637925 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.636113 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.637925 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.636130 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.638630 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.638609 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.640037 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.639151 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.640037 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.639612 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.649215 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.649191 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57xg5\" (UniqueName: \"kubernetes.io/projected/9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070-kube-api-access-57xg5\") pod \"prometheus-k8s-0\" (UID: \"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:51.777857 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:51.777836 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:52.087609 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:52.087582 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 20:56:52.090313 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:56:52.090285 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c54bb0f_91a3_4aa1_bdd4_86cf6d5bb070.slice/crio-0d6f38aa108ac151ed2542a8d4a3a152d96d35ce542a6e9263fbe2d15234b219 WatchSource:0}: Error finding container 0d6f38aa108ac151ed2542a8d4a3a152d96d35ce542a6e9263fbe2d15234b219: Status 404 returned error can't find the container with id 0d6f38aa108ac151ed2542a8d4a3a152d96d35ce542a6e9263fbe2d15234b219 Sep 09 20:56:52.348776 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:52.348718 2580 generic.go:356] "Generic (PLEG): container finished" podID="9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070" containerID="81ae4d84177fb6df367e922012cb94ae36cb49e97323d842c346b740baffeec8" exitCode=0 Sep 09 20:56:52.348776 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:52.348758 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070","Type":"ContainerDied","Data":"81ae4d84177fb6df367e922012cb94ae36cb49e97323d842c346b740baffeec8"} Sep 09 20:56:52.348884 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:52.348827 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070","Type":"ContainerStarted","Data":"0d6f38aa108ac151ed2542a8d4a3a152d96d35ce542a6e9263fbe2d15234b219"} Sep 09 20:56:53.042509 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:53.042478 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb3935b1-6307-4609-87ae-5c21a75a7530" path="/var/lib/kubelet/pods/bb3935b1-6307-4609-87ae-5c21a75a7530/volumes" Sep 09 20:56:53.355098 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:53.355039 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070","Type":"ContainerStarted","Data":"9d73cd0bb58d81f922e94a45762de9be1a1f4b2b1a4934a03d0742d048cbd625"} Sep 09 20:56:53.355504 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:53.355107 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070","Type":"ContainerStarted","Data":"58d30c438a5e6e0c4eb22a6bc9779a6e433f0abbefcbd97d3eca00407404de2f"} Sep 09 20:56:53.355504 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:53.355157 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070","Type":"ContainerStarted","Data":"8989b4bfa8b300b857f1947937746a44a55a5e8a493e619c546b49f2f01de551"} Sep 09 20:56:53.355504 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:53.355199 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070","Type":"ContainerStarted","Data":"f28a2a62d2f782375e20cd7b4b35704e830c98f6e2ac023c48ad163ffb52cc9b"} Sep 09 20:56:53.355504 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:53.355245 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070","Type":"ContainerStarted","Data":"8013a1b2686978a78cde1db4d82464d77c41540c083b85450114de5d4d36fa1d"} Sep 09 20:56:53.355504 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:53.355284 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070","Type":"ContainerStarted","Data":"7fff93c2e4620e4c70a25bc1e6ff586ec430e5b0b5fb9d24cc87064f39fa6ab9"} Sep 09 20:56:54.274511 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.274463 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.255180844 podStartE2EDuration="3.274444793s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.263949ms" isStatefulPod=false podCreationTimestamp="2026-09-09 20:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:56:53.413860588 +0000 UTC m=+390.874502511" watchObservedRunningTime="2026-09-09 20:56:54.274444793 +0000 UTC m=+391.735086717" Sep 09 20:56:54.275506 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.275490 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59d97fb867-v2bm2"] Sep 09 20:56:54.279854 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.279824 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:56:54.304148 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.304124 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59d97fb867-v2bm2"] Sep 09 20:56:54.354830 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.354801 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwpnq\" (UniqueName: \"kubernetes.io/projected/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-kube-api-access-mwpnq\") pod \"console-59d97fb867-v2bm2\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:56:54.354960 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.354835 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-console-oauth-config\") pod \"console-59d97fb867-v2bm2\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:56:54.354960 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.354854 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-trusted-ca-bundle\") pod \"console-59d97fb867-v2bm2\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:56:54.354960 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.354873 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-console-config\") pod \"console-59d97fb867-v2bm2\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:56:54.354960 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.354955 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-service-ca\") pod \"console-59d97fb867-v2bm2\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:56:54.355115 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.354982 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-console-serving-cert\") pod \"console-59d97fb867-v2bm2\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:56:54.355115 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.355003 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-oauth-serving-cert\") pod \"console-59d97fb867-v2bm2\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:56:54.404091 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.404061 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:56:54.456134 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.456114 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-console-config\") pod \"console-59d97fb867-v2bm2\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:56:54.456216 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.456178 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-service-ca\") pod \"console-59d97fb867-v2bm2\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:56:54.456529 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.456508 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-console-serving-cert\") pod \"console-59d97fb867-v2bm2\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:56:54.456618 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.456544 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-oauth-serving-cert\") pod \"console-59d97fb867-v2bm2\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:56:54.456676 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.456657 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwpnq\" (UniqueName: \"kubernetes.io/projected/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-kube-api-access-mwpnq\") pod \"console-59d97fb867-v2bm2\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:56:54.456779 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.456763 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-console-oauth-config\") pod \"console-59d97fb867-v2bm2\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:56:54.456843 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.456817 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-trusted-ca-bundle\") pod \"console-59d97fb867-v2bm2\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:56:54.456957 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.456934 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-service-ca\") pod \"console-59d97fb867-v2bm2\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:56:54.457018 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.456986 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-console-config\") pod \"console-59d97fb867-v2bm2\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:56:54.457172 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.457131 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-oauth-serving-cert\") pod \"console-59d97fb867-v2bm2\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:56:54.457879 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.457861 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-trusted-ca-bundle\") pod \"console-59d97fb867-v2bm2\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:56:54.459074 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.459056 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-console-serving-cert\") pod \"console-59d97fb867-v2bm2\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:56:54.459563 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.459543 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-console-oauth-config\") pod \"console-59d97fb867-v2bm2\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:56:54.468794 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.468774 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwpnq\" (UniqueName: \"kubernetes.io/projected/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-kube-api-access-mwpnq\") pod \"console-59d97fb867-v2bm2\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:56:54.504186 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.504162 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:56:54.588951 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.588878 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:56:54.952270 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:54.952244 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59d97fb867-v2bm2"] Sep 09 20:56:54.954357 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:56:54.954332 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf3d3c84_46fd_4cd6_b83b_ab2b61771875.slice/crio-272cf263dc873ea9f8f8d950632afa161c78d0ff2dcb04f056000440a4877ae6 WatchSource:0}: Error finding container 272cf263dc873ea9f8f8d950632afa161c78d0ff2dcb04f056000440a4877ae6: Status 404 returned error can't find the container with id 272cf263dc873ea9f8f8d950632afa161c78d0ff2dcb04f056000440a4877ae6 Sep 09 20:56:55.363871 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:55.363800 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d97fb867-v2bm2" event={"ID":"bf3d3c84-46fd-4cd6-b83b-ab2b61771875","Type":"ContainerStarted","Data":"f1d7a227988994766efdd87826b1b48eb909785f24904d1ff62b0096213ef0d3"} Sep 09 20:56:55.363871 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:55.363847 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d97fb867-v2bm2" event={"ID":"bf3d3c84-46fd-4cd6-b83b-ab2b61771875","Type":"ContainerStarted","Data":"272cf263dc873ea9f8f8d950632afa161c78d0ff2dcb04f056000440a4877ae6"} Sep 09 20:56:55.403313 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:55.403247 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59d97fb867-v2bm2" podStartSLOduration=1.403227759 podStartE2EDuration="1.403227759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:56:55.402284171 +0000 UTC m=+392.862926093" watchObservedRunningTime="2026-09-09 20:56:55.403227759 +0000 UTC m=+392.863869682" Sep 09 20:56:56.778179 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:56:56.778148 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:57:04.589945 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:04.589914 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:57:04.590287 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:04.589969 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:57:04.594650 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:04.594628 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:57:05.403458 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:05.403432 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:57:05.450327 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:05.450299 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54684f4fb4-vlcsw"] Sep 09 20:57:05.590194 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:05.590158 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:57:05.685930 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:05.685830 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:57:05.785288 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:05.785266 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:57:05.886193 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:05.886173 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:57:30.470610 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:30.470556 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54684f4fb4-vlcsw" podUID="71ef8c0e-3bfe-4b8a-845e-d432e173d3a8" containerName="console" containerID="cri-o://637842429c53e46136d60f721b92fcd4657c69d438e05747b0641dbfa3ce3f7f" gracePeriod=15 Sep 09 20:57:30.701402 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:30.701380 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:57:30.799617 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:30.799562 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6x4kh\" (UniqueName: \"kubernetes.io/projected/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-kube-api-access-6x4kh\") pod \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " Sep 09 20:57:30.799617 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:30.799591 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-service-ca\") pod \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " Sep 09 20:57:30.799617 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:30.799616 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-console-oauth-config\") pod \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " Sep 09 20:57:30.799809 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:30.799660 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-oauth-serving-cert\") pod \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " Sep 09 20:57:30.799809 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:30.799683 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-trusted-ca-bundle\") pod \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " Sep 09 20:57:30.799809 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:30.799699 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-console-serving-cert\") pod \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " Sep 09 20:57:30.799809 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:30.799745 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-console-config\") pod \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\" (UID: \"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8\") " Sep 09 20:57:30.800116 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:30.800073 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-service-ca" (OuterVolumeSpecName: "service-ca") pod "71ef8c0e-3bfe-4b8a-845e-d432e173d3a8" (UID: "71ef8c0e-3bfe-4b8a-845e-d432e173d3a8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:57:30.800224 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:30.800177 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "71ef8c0e-3bfe-4b8a-845e-d432e173d3a8" (UID: "71ef8c0e-3bfe-4b8a-845e-d432e173d3a8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:57:30.800224 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:30.800192 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-console-config" (OuterVolumeSpecName: "console-config") pod "71ef8c0e-3bfe-4b8a-845e-d432e173d3a8" (UID: "71ef8c0e-3bfe-4b8a-845e-d432e173d3a8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:57:30.800224 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:30.800232 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "71ef8c0e-3bfe-4b8a-845e-d432e173d3a8" (UID: "71ef8c0e-3bfe-4b8a-845e-d432e173d3a8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:57:30.801768 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:30.801746 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-kube-api-access-6x4kh" (OuterVolumeSpecName: "kube-api-access-6x4kh") pod "71ef8c0e-3bfe-4b8a-845e-d432e173d3a8" (UID: "71ef8c0e-3bfe-4b8a-845e-d432e173d3a8"). InnerVolumeSpecName "kube-api-access-6x4kh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:57:30.801845 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:30.801782 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "71ef8c0e-3bfe-4b8a-845e-d432e173d3a8" (UID: "71ef8c0e-3bfe-4b8a-845e-d432e173d3a8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:57:30.801881 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:30.801844 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "71ef8c0e-3bfe-4b8a-845e-d432e173d3a8" (UID: "71ef8c0e-3bfe-4b8a-845e-d432e173d3a8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:57:30.900436 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:30.900412 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x4kh\" (UniqueName: \"kubernetes.io/projected/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-kube-api-access-6x4kh\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:57:30.900436 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:30.900435 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-service-ca\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:57:30.900548 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:30.900444 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-console-oauth-config\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:57:30.900548 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:30.900453 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-oauth-serving-cert\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:57:30.900548 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:30.900461 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-trusted-ca-bundle\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:57:30.900548 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:30.900469 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-console-serving-cert\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:57:30.900548 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:30.900477 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8-console-config\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:57:31.482232 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:31.482201 2580 generic.go:356] "Generic (PLEG): container finished" podID="71ef8c0e-3bfe-4b8a-845e-d432e173d3a8" containerID="637842429c53e46136d60f721b92fcd4657c69d438e05747b0641dbfa3ce3f7f" exitCode=0 Sep 09 20:57:31.482597 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:31.482246 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54684f4fb4-vlcsw" Sep 09 20:57:31.482597 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:31.482270 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54684f4fb4-vlcsw" event={"ID":"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8","Type":"ContainerDied","Data":"637842429c53e46136d60f721b92fcd4657c69d438e05747b0641dbfa3ce3f7f"} Sep 09 20:57:31.482597 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:31.482318 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54684f4fb4-vlcsw" event={"ID":"71ef8c0e-3bfe-4b8a-845e-d432e173d3a8","Type":"ContainerDied","Data":"06bb57ee0a10a16f5e322e35e59d6797f0677baa1169f46894a9ea7b934b9854"} Sep 09 20:57:31.482597 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:31.482347 2580 scope.go:117] "RemoveContainer" containerID="637842429c53e46136d60f721b92fcd4657c69d438e05747b0641dbfa3ce3f7f" Sep 09 20:57:31.489857 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:31.489842 2580 scope.go:117] "RemoveContainer" containerID="637842429c53e46136d60f721b92fcd4657c69d438e05747b0641dbfa3ce3f7f" Sep 09 20:57:31.490146 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:57:31.490128 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"637842429c53e46136d60f721b92fcd4657c69d438e05747b0641dbfa3ce3f7f\": container with ID starting with 637842429c53e46136d60f721b92fcd4657c69d438e05747b0641dbfa3ce3f7f not found: ID does not exist" containerID="637842429c53e46136d60f721b92fcd4657c69d438e05747b0641dbfa3ce3f7f" Sep 09 20:57:31.490212 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:31.490166 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"637842429c53e46136d60f721b92fcd4657c69d438e05747b0641dbfa3ce3f7f"} err="failed to get container status \"637842429c53e46136d60f721b92fcd4657c69d438e05747b0641dbfa3ce3f7f\": rpc error: code = NotFound desc = could not find container \"637842429c53e46136d60f721b92fcd4657c69d438e05747b0641dbfa3ce3f7f\": container with ID starting with 637842429c53e46136d60f721b92fcd4657c69d438e05747b0641dbfa3ce3f7f not found: ID does not exist" Sep 09 20:57:31.507441 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:31.507422 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54684f4fb4-vlcsw"] Sep 09 20:57:31.512808 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:31.512789 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54684f4fb4-vlcsw"] Sep 09 20:57:33.041360 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:33.041332 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71ef8c0e-3bfe-4b8a-845e-d432e173d3a8" path="/var/lib/kubelet/pods/71ef8c0e-3bfe-4b8a-845e-d432e173d3a8/volumes" Sep 09 20:57:38.427553 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:38.427519 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:57:38.495015 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:38.494986 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:57:51.778191 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:51.778105 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:57:51.793113 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:51.793088 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:57:52.565436 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:57:52.565413 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:58:00.583436 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:00.583399 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:00.651812 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:00.651786 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:00.931735 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:00.931635 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:01.176145 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:01.176100 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:03.703070 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.703041 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8c7bcd8d4-vzfdp"] Sep 09 20:58:03.703596 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.703364 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71ef8c0e-3bfe-4b8a-845e-d432e173d3a8" containerName="console" Sep 09 20:58:03.703596 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.703381 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="71ef8c0e-3bfe-4b8a-845e-d432e173d3a8" containerName="console" Sep 09 20:58:03.703596 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.703450 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="71ef8c0e-3bfe-4b8a-845e-d432e173d3a8" containerName="console" Sep 09 20:58:03.707571 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.707545 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:03.727368 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.727338 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8c7bcd8d4-vzfdp"] Sep 09 20:58:03.773546 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.773512 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:03.811412 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.811387 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72bb276c-bd16-47d6-a429-58a41d4302bb-console-config\") pod \"console-8c7bcd8d4-vzfdp\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:03.811485 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.811422 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72bb276c-bd16-47d6-a429-58a41d4302bb-console-serving-cert\") pod \"console-8c7bcd8d4-vzfdp\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:03.811485 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.811445 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72bb276c-bd16-47d6-a429-58a41d4302bb-trusted-ca-bundle\") pod \"console-8c7bcd8d4-vzfdp\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:03.811485 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.811466 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72bb276c-bd16-47d6-a429-58a41d4302bb-service-ca\") pod \"console-8c7bcd8d4-vzfdp\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:03.811580 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.811521 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72bb276c-bd16-47d6-a429-58a41d4302bb-console-oauth-config\") pod \"console-8c7bcd8d4-vzfdp\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:03.811580 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.811536 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72bb276c-bd16-47d6-a429-58a41d4302bb-oauth-serving-cert\") pod \"console-8c7bcd8d4-vzfdp\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:03.811638 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.811598 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz954\" (UniqueName: \"kubernetes.io/projected/72bb276c-bd16-47d6-a429-58a41d4302bb-kube-api-access-jz954\") pod \"console-8c7bcd8d4-vzfdp\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:03.910474 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.910450 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:03.912214 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.912197 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72bb276c-bd16-47d6-a429-58a41d4302bb-console-serving-cert\") pod \"console-8c7bcd8d4-vzfdp\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:03.912284 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.912221 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72bb276c-bd16-47d6-a429-58a41d4302bb-trusted-ca-bundle\") pod \"console-8c7bcd8d4-vzfdp\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:03.912284 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.912242 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72bb276c-bd16-47d6-a429-58a41d4302bb-service-ca\") pod \"console-8c7bcd8d4-vzfdp\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:03.912284 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.912265 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72bb276c-bd16-47d6-a429-58a41d4302bb-console-oauth-config\") pod \"console-8c7bcd8d4-vzfdp\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:03.912284 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.912280 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72bb276c-bd16-47d6-a429-58a41d4302bb-oauth-serving-cert\") pod \"console-8c7bcd8d4-vzfdp\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:03.912472 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.912319 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz954\" (UniqueName: \"kubernetes.io/projected/72bb276c-bd16-47d6-a429-58a41d4302bb-kube-api-access-jz954\") pod \"console-8c7bcd8d4-vzfdp\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:03.912472 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.912350 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72bb276c-bd16-47d6-a429-58a41d4302bb-console-config\") pod \"console-8c7bcd8d4-vzfdp\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:03.913112 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.913015 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72bb276c-bd16-47d6-a429-58a41d4302bb-service-ca\") pod \"console-8c7bcd8d4-vzfdp\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:03.913112 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.913064 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72bb276c-bd16-47d6-a429-58a41d4302bb-console-config\") pod \"console-8c7bcd8d4-vzfdp\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:03.913276 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.913259 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72bb276c-bd16-47d6-a429-58a41d4302bb-trusted-ca-bundle\") pod \"console-8c7bcd8d4-vzfdp\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:03.913341 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.913301 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72bb276c-bd16-47d6-a429-58a41d4302bb-oauth-serving-cert\") pod \"console-8c7bcd8d4-vzfdp\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:03.914766 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.914746 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72bb276c-bd16-47d6-a429-58a41d4302bb-console-serving-cert\") pod \"console-8c7bcd8d4-vzfdp\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:03.914854 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.914767 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72bb276c-bd16-47d6-a429-58a41d4302bb-console-oauth-config\") pod \"console-8c7bcd8d4-vzfdp\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:03.923376 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:03.923356 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz954\" (UniqueName: \"kubernetes.io/projected/72bb276c-bd16-47d6-a429-58a41d4302bb-kube-api-access-jz954\") pod \"console-8c7bcd8d4-vzfdp\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:04.016013 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:04.015964 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:04.292272 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:04.292206 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8c7bcd8d4-vzfdp"] Sep 09 20:58:04.295170 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:58:04.295140 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72bb276c_bd16_47d6_a429_58a41d4302bb.slice/crio-b398be1082f2d95e74f8be1d16b20a0aa205beec31030859c93dcb1693946a2c WatchSource:0}: Error finding container b398be1082f2d95e74f8be1d16b20a0aa205beec31030859c93dcb1693946a2c: Status 404 returned error can't find the container with id b398be1082f2d95e74f8be1d16b20a0aa205beec31030859c93dcb1693946a2c Sep 09 20:58:04.591015 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:04.590925 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8c7bcd8d4-vzfdp" event={"ID":"72bb276c-bd16-47d6-a429-58a41d4302bb","Type":"ContainerStarted","Data":"2c5c1076e7028ca3f94f97e4b6dbf1b2bcc614321882d283b01df85b510ad877"} Sep 09 20:58:04.591015 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:04.590972 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8c7bcd8d4-vzfdp" event={"ID":"72bb276c-bd16-47d6-a429-58a41d4302bb","Type":"ContainerStarted","Data":"b398be1082f2d95e74f8be1d16b20a0aa205beec31030859c93dcb1693946a2c"} Sep 09 20:58:04.618053 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:04.617996 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8c7bcd8d4-vzfdp" podStartSLOduration=1.617978467 podStartE2EDuration="1.617978467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:04.615933315 +0000 UTC m=+462.076575238" watchObservedRunningTime="2026-09-09 20:58:04.617978467 +0000 UTC m=+462.078620387" Sep 09 20:58:12.941333 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:12.941302 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:13.074473 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:13.074438 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:14.016461 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:14.016434 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:14.016807 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:14.016500 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:14.020746 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:14.020728 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:14.631361 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:14.631336 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 20:58:14.776899 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:14.776856 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59d97fb867-v2bm2"] Sep 09 20:58:15.033175 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:15.033146 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:15.157340 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:15.157315 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:26.725956 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:26.725918 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:26.824733 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:26.824704 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:39.794997 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:39.794931 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59d97fb867-v2bm2" podUID="bf3d3c84-46fd-4cd6-b83b-ab2b61771875" containerName="console" containerID="cri-o://f1d7a227988994766efdd87826b1b48eb909785f24904d1ff62b0096213ef0d3" gracePeriod=15 Sep 09 20:58:40.027344 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.027319 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:58:40.059759 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.059706 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-trusted-ca-bundle\") pod \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " Sep 09 20:58:40.059759 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.059738 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-oauth-serving-cert\") pod \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " Sep 09 20:58:40.059759 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.059757 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-console-config\") pod \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " Sep 09 20:58:40.060012 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.059797 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-console-oauth-config\") pod \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " Sep 09 20:58:40.060012 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.059840 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwpnq\" (UniqueName: \"kubernetes.io/projected/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-kube-api-access-mwpnq\") pod \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " Sep 09 20:58:40.060012 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.059875 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-service-ca\") pod \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " Sep 09 20:58:40.060012 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.059933 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-console-serving-cert\") pod \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\" (UID: \"bf3d3c84-46fd-4cd6-b83b-ab2b61771875\") " Sep 09 20:58:40.060208 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.060181 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-console-config" (OuterVolumeSpecName: "console-config") pod "bf3d3c84-46fd-4cd6-b83b-ab2b61771875" (UID: "bf3d3c84-46fd-4cd6-b83b-ab2b61771875"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:58:40.060254 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.060199 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bf3d3c84-46fd-4cd6-b83b-ab2b61771875" (UID: "bf3d3c84-46fd-4cd6-b83b-ab2b61771875"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:58:40.060254 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.060238 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bf3d3c84-46fd-4cd6-b83b-ab2b61771875" (UID: "bf3d3c84-46fd-4cd6-b83b-ab2b61771875"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:58:40.060332 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.060286 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-service-ca" (OuterVolumeSpecName: "service-ca") pod "bf3d3c84-46fd-4cd6-b83b-ab2b61771875" (UID: "bf3d3c84-46fd-4cd6-b83b-ab2b61771875"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:58:40.062169 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.062141 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bf3d3c84-46fd-4cd6-b83b-ab2b61771875" (UID: "bf3d3c84-46fd-4cd6-b83b-ab2b61771875"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:58:40.062266 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.062192 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bf3d3c84-46fd-4cd6-b83b-ab2b61771875" (UID: "bf3d3c84-46fd-4cd6-b83b-ab2b61771875"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:58:40.062266 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.062201 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-kube-api-access-mwpnq" (OuterVolumeSpecName: "kube-api-access-mwpnq") pod "bf3d3c84-46fd-4cd6-b83b-ab2b61771875" (UID: "bf3d3c84-46fd-4cd6-b83b-ab2b61771875"). InnerVolumeSpecName "kube-api-access-mwpnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:58:40.160997 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.160969 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-console-oauth-config\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:58:40.160997 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.160992 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwpnq\" (UniqueName: \"kubernetes.io/projected/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-kube-api-access-mwpnq\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:58:40.160997 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.161001 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-service-ca\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:58:40.161190 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.161010 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-console-serving-cert\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:58:40.161190 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.161018 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-trusted-ca-bundle\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:58:40.161190 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.161026 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-oauth-serving-cert\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:58:40.161190 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.161034 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf3d3c84-46fd-4cd6-b83b-ab2b61771875-console-config\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 20:58:40.712906 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.712867 2580 generic.go:356] "Generic (PLEG): container finished" podID="bf3d3c84-46fd-4cd6-b83b-ab2b61771875" containerID="f1d7a227988994766efdd87826b1b48eb909785f24904d1ff62b0096213ef0d3" exitCode=0 Sep 09 20:58:40.713097 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.712944 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d97fb867-v2bm2" event={"ID":"bf3d3c84-46fd-4cd6-b83b-ab2b61771875","Type":"ContainerDied","Data":"f1d7a227988994766efdd87826b1b48eb909785f24904d1ff62b0096213ef0d3"} Sep 09 20:58:40.713097 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.712953 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d97fb867-v2bm2" Sep 09 20:58:40.713097 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.713008 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d97fb867-v2bm2" event={"ID":"bf3d3c84-46fd-4cd6-b83b-ab2b61771875","Type":"ContainerDied","Data":"272cf263dc873ea9f8f8d950632afa161c78d0ff2dcb04f056000440a4877ae6"} Sep 09 20:58:40.713097 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.713027 2580 scope.go:117] "RemoveContainer" containerID="f1d7a227988994766efdd87826b1b48eb909785f24904d1ff62b0096213ef0d3" Sep 09 20:58:40.721013 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.720996 2580 scope.go:117] "RemoveContainer" containerID="f1d7a227988994766efdd87826b1b48eb909785f24904d1ff62b0096213ef0d3" Sep 09 20:58:40.721277 ip-10-0-137-19 kubenswrapper[2580]: E0909 20:58:40.721258 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1d7a227988994766efdd87826b1b48eb909785f24904d1ff62b0096213ef0d3\": container with ID starting with f1d7a227988994766efdd87826b1b48eb909785f24904d1ff62b0096213ef0d3 not found: ID does not exist" containerID="f1d7a227988994766efdd87826b1b48eb909785f24904d1ff62b0096213ef0d3" Sep 09 20:58:40.721315 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.721286 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d7a227988994766efdd87826b1b48eb909785f24904d1ff62b0096213ef0d3"} err="failed to get container status \"f1d7a227988994766efdd87826b1b48eb909785f24904d1ff62b0096213ef0d3\": rpc error: code = NotFound desc = could not find container \"f1d7a227988994766efdd87826b1b48eb909785f24904d1ff62b0096213ef0d3\": container with ID starting with f1d7a227988994766efdd87826b1b48eb909785f24904d1ff62b0096213ef0d3 not found: ID does not exist" Sep 09 20:58:40.733923 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.733881 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59d97fb867-v2bm2"] Sep 09 20:58:40.735938 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:40.735917 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59d97fb867-v2bm2"] Sep 09 20:58:41.042224 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:41.042140 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf3d3c84-46fd-4cd6-b83b-ab2b61771875" path="/var/lib/kubelet/pods/bf3d3c84-46fd-4cd6-b83b-ab2b61771875/volumes" Sep 09 20:58:53.588136 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:53.588090 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:53.681046 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:58:53.681019 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:15.736152 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:15.736055 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:16.070106 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:16.070036 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:20.040005 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:20.039959 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:20.214798 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:20.214773 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:21.609753 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:21.609724 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:21.919312 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:21.919242 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:22.194827 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:22.194801 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:24.268103 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:24.268047 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:24.772193 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:24.772163 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:26.597146 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:26.597104 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:26.760972 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:26.760943 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:32.021571 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:32.021541 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:33.203581 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:33.203536 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:33.340062 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:33.340037 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:33.939004 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:33.938971 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:34.036210 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:34.036161 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:38.944382 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:38.944344 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:39.082287 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:39.082258 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:40.860628 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:40.860580 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:52.622610 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:52.622577 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:52.751294 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:52.751252 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:53.271341 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:53.271306 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:53.936737 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:53.936705 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:54.115876 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.115846 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4"] Sep 09 20:59:54.116130 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.116118 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf3d3c84-46fd-4cd6-b83b-ab2b61771875" containerName="console" Sep 09 20:59:54.116175 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.116132 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf3d3c84-46fd-4cd6-b83b-ab2b61771875" containerName="console" Sep 09 20:59:54.116216 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.116176 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf3d3c84-46fd-4cd6-b83b-ab2b61771875" containerName="console" Sep 09 20:59:54.118923 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.118909 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4" Sep 09 20:59:54.123043 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.123023 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 09 20:59:54.123254 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.123240 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 09 20:59:54.123469 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.123456 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 09 20:59:54.123524 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.123458 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-v47mq\"" Sep 09 20:59:54.123765 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.123744 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 09 20:59:54.128630 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.128607 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:54.139421 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.139404 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4"] Sep 09 20:59:54.247084 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.247027 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a43b1176-2185-44bd-acf9-5b24408e9095-manifests\") pod \"opendatahub-operator-controller-manager-cf9547dff-vpfp4\" (UID: \"a43b1176-2185-44bd-acf9-5b24408e9095\") " pod="opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4" Sep 09 20:59:54.247174 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.247084 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxszl\" (UniqueName: \"kubernetes.io/projected/a43b1176-2185-44bd-acf9-5b24408e9095-kube-api-access-gxszl\") pod \"opendatahub-operator-controller-manager-cf9547dff-vpfp4\" (UID: \"a43b1176-2185-44bd-acf9-5b24408e9095\") " pod="opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4" Sep 09 20:59:54.247174 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.247134 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a43b1176-2185-44bd-acf9-5b24408e9095-tmp\") pod \"opendatahub-operator-controller-manager-cf9547dff-vpfp4\" (UID: \"a43b1176-2185-44bd-acf9-5b24408e9095\") " pod="opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4" Sep 09 20:59:54.247274 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.247173 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a43b1176-2185-44bd-acf9-5b24408e9095-webhook-cert\") pod \"opendatahub-operator-controller-manager-cf9547dff-vpfp4\" (UID: \"a43b1176-2185-44bd-acf9-5b24408e9095\") " pod="opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4" Sep 09 20:59:54.247274 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.247230 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a43b1176-2185-44bd-acf9-5b24408e9095-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cf9547dff-vpfp4\" (UID: \"a43b1176-2185-44bd-acf9-5b24408e9095\") " pod="opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4" Sep 09 20:59:54.281158 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.281138 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:54.348442 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.348419 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxszl\" (UniqueName: \"kubernetes.io/projected/a43b1176-2185-44bd-acf9-5b24408e9095-kube-api-access-gxszl\") pod \"opendatahub-operator-controller-manager-cf9547dff-vpfp4\" (UID: \"a43b1176-2185-44bd-acf9-5b24408e9095\") " pod="opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4" Sep 09 20:59:54.348526 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.348447 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a43b1176-2185-44bd-acf9-5b24408e9095-tmp\") pod \"opendatahub-operator-controller-manager-cf9547dff-vpfp4\" (UID: \"a43b1176-2185-44bd-acf9-5b24408e9095\") " pod="opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4" Sep 09 20:59:54.348526 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.348464 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a43b1176-2185-44bd-acf9-5b24408e9095-webhook-cert\") pod \"opendatahub-operator-controller-manager-cf9547dff-vpfp4\" (UID: \"a43b1176-2185-44bd-acf9-5b24408e9095\") " pod="opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4" Sep 09 20:59:54.348526 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.348485 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a43b1176-2185-44bd-acf9-5b24408e9095-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cf9547dff-vpfp4\" (UID: \"a43b1176-2185-44bd-acf9-5b24408e9095\") " pod="opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4" Sep 09 20:59:54.348526 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.348510 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a43b1176-2185-44bd-acf9-5b24408e9095-manifests\") pod \"opendatahub-operator-controller-manager-cf9547dff-vpfp4\" (UID: \"a43b1176-2185-44bd-acf9-5b24408e9095\") " pod="opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4" Sep 09 20:59:54.348822 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.348807 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a43b1176-2185-44bd-acf9-5b24408e9095-manifests\") pod \"opendatahub-operator-controller-manager-cf9547dff-vpfp4\" (UID: \"a43b1176-2185-44bd-acf9-5b24408e9095\") " pod="opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4" Sep 09 20:59:54.350671 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.350652 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a43b1176-2185-44bd-acf9-5b24408e9095-tmp\") pod \"opendatahub-operator-controller-manager-cf9547dff-vpfp4\" (UID: \"a43b1176-2185-44bd-acf9-5b24408e9095\") " pod="opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4" Sep 09 20:59:54.350940 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.350923 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a43b1176-2185-44bd-acf9-5b24408e9095-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cf9547dff-vpfp4\" (UID: \"a43b1176-2185-44bd-acf9-5b24408e9095\") " pod="opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4" Sep 09 20:59:54.351012 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.350964 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a43b1176-2185-44bd-acf9-5b24408e9095-webhook-cert\") pod \"opendatahub-operator-controller-manager-cf9547dff-vpfp4\" (UID: \"a43b1176-2185-44bd-acf9-5b24408e9095\") " pod="opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4" Sep 09 20:59:54.362224 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.362195 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxszl\" (UniqueName: \"kubernetes.io/projected/a43b1176-2185-44bd-acf9-5b24408e9095-kube-api-access-gxszl\") pod \"opendatahub-operator-controller-manager-cf9547dff-vpfp4\" (UID: \"a43b1176-2185-44bd-acf9-5b24408e9095\") " pod="opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4" Sep 09 20:59:54.401604 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.401582 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:54.428636 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.428617 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4" Sep 09 20:59:54.514015 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.513955 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:54.624948 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.624919 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:54.720507 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.720408 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4"] Sep 09 20:59:54.722584 ip-10-0-137-19 kubenswrapper[2580]: W0909 20:59:54.722563 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43b1176_2185_44bd_acf9_5b24408e9095.slice/crio-bfa82603f61e115342c1f44b454b7f04e75fdca3e65a1eea5700feefe6caa86d WatchSource:0}: Error finding container bfa82603f61e115342c1f44b454b7f04e75fdca3e65a1eea5700feefe6caa86d: Status 404 returned error can't find the container with id bfa82603f61e115342c1f44b454b7f04e75fdca3e65a1eea5700feefe6caa86d Sep 09 20:59:54.792625 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.792568 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:54.950583 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.950555 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4" event={"ID":"a43b1176-2185-44bd-acf9-5b24408e9095","Type":"ContainerStarted","Data":"bfa82603f61e115342c1f44b454b7f04e75fdca3e65a1eea5700feefe6caa86d"} Sep 09 20:59:54.965039 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:54.965016 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:57.964587 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:57.964558 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4" event={"ID":"a43b1176-2185-44bd-acf9-5b24408e9095","Type":"ContainerStarted","Data":"ffe589c61212e5d97a597116a7c2f89bcd6042a86c0088cce3ec11027a440231"} Sep 09 20:59:58.969629 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:58.969594 2580 generic.go:356] "Generic (PLEG): container finished" podID="a43b1176-2185-44bd-acf9-5b24408e9095" containerID="ffe589c61212e5d97a597116a7c2f89bcd6042a86c0088cce3ec11027a440231" exitCode=0 Sep 09 20:59:58.969963 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:58.969676 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4" event={"ID":"a43b1176-2185-44bd-acf9-5b24408e9095","Type":"ContainerDied","Data":"ffe589c61212e5d97a597116a7c2f89bcd6042a86c0088cce3ec11027a440231"} Sep 09 20:59:59.974868 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:59.974834 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4" event={"ID":"a43b1176-2185-44bd-acf9-5b24408e9095","Type":"ContainerStarted","Data":"501a7669fbb9520c3d589a569aefd4350effa3e46dfd303353a2d4054aa4809d"} Sep 09 20:59:59.975242 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:59.974952 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4" Sep 09 20:59:59.998548 ip-10-0-137-19 kubenswrapper[2580]: I0909 20:59:59.998497 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4" podStartSLOduration=1.9096653190000001 podStartE2EDuration="5.998480155s" totalImagesPullingTime="3.177434477s" totalInitContainerRuntime="911.380359ms" isStatefulPod=false podCreationTimestamp="2026-09-09 20:59:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:59:59.997779016 +0000 UTC m=+577.458420938" watchObservedRunningTime="2026-09-09 20:59:59.998480155 +0000 UTC m=+577.459122076" Sep 09 21:00:03.197369 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:03.197340 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:05.838416 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:05.838387 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:10.980714 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:10.980681 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-cf9547dff-vpfp4" Sep 09 21:00:13.264714 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:13.264683 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:13.399870 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:13.399816 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:13.664291 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:13.664172 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:14.172119 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:14.172083 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:15.637576 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:15.637541 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:15.718490 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:15.718450 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:17.174699 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:17.174654 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:19.929706 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:19.929673 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:20.030557 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:20.030535 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:20.622087 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:20.622055 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:20.896330 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:20.896251 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:21.113833 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:21.113784 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:22.967407 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:22.967375 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tctqv_ddb3d74a-a1bb-4bda-8d0d-89320e50f357/ovn-acl-logging/0.log" Sep 09 21:00:22.969222 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:22.969204 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tctqv_ddb3d74a-a1bb-4bda-8d0d-89320e50f357/ovn-acl-logging/0.log" Sep 09 21:00:24.458751 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:24.458722 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:24.664420 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:24.664387 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:24.795079 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:24.795008 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:25.097710 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:25.097624 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:25.421043 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:25.420958 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:27.818015 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:27.817978 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-789f7f47d6-tcbm2"] Sep 09 21:00:27.821770 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:27.821747 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:27.844330 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:27.844305 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-789f7f47d6-tcbm2"] Sep 09 21:00:27.881908 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:27.881861 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68724704-7c68-4fdf-9a76-b20b095de703-console-config\") pod \"console-789f7f47d6-tcbm2\" (UID: \"68724704-7c68-4fdf-9a76-b20b095de703\") " pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:27.882036 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:27.881963 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwd46\" (UniqueName: \"kubernetes.io/projected/68724704-7c68-4fdf-9a76-b20b095de703-kube-api-access-dwd46\") pod \"console-789f7f47d6-tcbm2\" (UID: \"68724704-7c68-4fdf-9a76-b20b095de703\") " pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:27.882036 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:27.882021 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68724704-7c68-4fdf-9a76-b20b095de703-oauth-serving-cert\") pod \"console-789f7f47d6-tcbm2\" (UID: \"68724704-7c68-4fdf-9a76-b20b095de703\") " pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:27.882129 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:27.882051 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68724704-7c68-4fdf-9a76-b20b095de703-service-ca\") pod \"console-789f7f47d6-tcbm2\" (UID: \"68724704-7c68-4fdf-9a76-b20b095de703\") " pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:27.882129 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:27.882074 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68724704-7c68-4fdf-9a76-b20b095de703-trusted-ca-bundle\") pod \"console-789f7f47d6-tcbm2\" (UID: \"68724704-7c68-4fdf-9a76-b20b095de703\") " pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:27.882208 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:27.882145 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68724704-7c68-4fdf-9a76-b20b095de703-console-serving-cert\") pod \"console-789f7f47d6-tcbm2\" (UID: \"68724704-7c68-4fdf-9a76-b20b095de703\") " pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:27.882208 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:27.882190 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68724704-7c68-4fdf-9a76-b20b095de703-console-oauth-config\") pod \"console-789f7f47d6-tcbm2\" (UID: \"68724704-7c68-4fdf-9a76-b20b095de703\") " pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:27.935325 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:27.935301 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:27.983340 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:27.983307 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68724704-7c68-4fdf-9a76-b20b095de703-service-ca\") pod \"console-789f7f47d6-tcbm2\" (UID: \"68724704-7c68-4fdf-9a76-b20b095de703\") " pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:27.983340 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:27.983339 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68724704-7c68-4fdf-9a76-b20b095de703-trusted-ca-bundle\") pod \"console-789f7f47d6-tcbm2\" (UID: \"68724704-7c68-4fdf-9a76-b20b095de703\") " pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:27.983582 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:27.983461 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68724704-7c68-4fdf-9a76-b20b095de703-console-serving-cert\") pod \"console-789f7f47d6-tcbm2\" (UID: \"68724704-7c68-4fdf-9a76-b20b095de703\") " pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:27.983582 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:27.983496 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68724704-7c68-4fdf-9a76-b20b095de703-console-oauth-config\") pod \"console-789f7f47d6-tcbm2\" (UID: \"68724704-7c68-4fdf-9a76-b20b095de703\") " pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:27.983582 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:27.983572 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68724704-7c68-4fdf-9a76-b20b095de703-console-config\") pod \"console-789f7f47d6-tcbm2\" (UID: \"68724704-7c68-4fdf-9a76-b20b095de703\") " pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:27.983738 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:27.983619 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwd46\" (UniqueName: \"kubernetes.io/projected/68724704-7c68-4fdf-9a76-b20b095de703-kube-api-access-dwd46\") pod \"console-789f7f47d6-tcbm2\" (UID: \"68724704-7c68-4fdf-9a76-b20b095de703\") " pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:27.983738 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:27.983659 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68724704-7c68-4fdf-9a76-b20b095de703-oauth-serving-cert\") pod \"console-789f7f47d6-tcbm2\" (UID: \"68724704-7c68-4fdf-9a76-b20b095de703\") " pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:27.984087 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:27.984061 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68724704-7c68-4fdf-9a76-b20b095de703-service-ca\") pod \"console-789f7f47d6-tcbm2\" (UID: \"68724704-7c68-4fdf-9a76-b20b095de703\") " pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:27.984264 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:27.984242 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68724704-7c68-4fdf-9a76-b20b095de703-console-config\") pod \"console-789f7f47d6-tcbm2\" (UID: \"68724704-7c68-4fdf-9a76-b20b095de703\") " pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:27.984264 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:27.984258 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68724704-7c68-4fdf-9a76-b20b095de703-trusted-ca-bundle\") pod \"console-789f7f47d6-tcbm2\" (UID: \"68724704-7c68-4fdf-9a76-b20b095de703\") " pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:27.984497 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:27.984478 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68724704-7c68-4fdf-9a76-b20b095de703-oauth-serving-cert\") pod \"console-789f7f47d6-tcbm2\" (UID: \"68724704-7c68-4fdf-9a76-b20b095de703\") " pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:27.986100 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:27.986079 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68724704-7c68-4fdf-9a76-b20b095de703-console-serving-cert\") pod \"console-789f7f47d6-tcbm2\" (UID: \"68724704-7c68-4fdf-9a76-b20b095de703\") " pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:27.986362 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:27.986343 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68724704-7c68-4fdf-9a76-b20b095de703-console-oauth-config\") pod \"console-789f7f47d6-tcbm2\" (UID: \"68724704-7c68-4fdf-9a76-b20b095de703\") " pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:27.992914 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:27.992869 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwd46\" (UniqueName: \"kubernetes.io/projected/68724704-7c68-4fdf-9a76-b20b095de703-kube-api-access-dwd46\") pod \"console-789f7f47d6-tcbm2\" (UID: \"68724704-7c68-4fdf-9a76-b20b095de703\") " pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:28.037055 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:28.037023 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:28.133502 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:28.133404 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:28.406195 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:28.406116 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-789f7f47d6-tcbm2"] Sep 09 21:00:28.409543 ip-10-0-137-19 kubenswrapper[2580]: W0909 21:00:28.409517 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68724704_7c68_4fdf_9a76_b20b095de703.slice/crio-ed9b5855abda565a57b52a8312b8f78b54b39155c71d8883a80a7ba3274dbe0c WatchSource:0}: Error finding container ed9b5855abda565a57b52a8312b8f78b54b39155c71d8883a80a7ba3274dbe0c: Status 404 returned error can't find the container with id ed9b5855abda565a57b52a8312b8f78b54b39155c71d8883a80a7ba3274dbe0c Sep 09 21:00:28.597595 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:28.597557 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:28.696189 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:28.696152 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:29.077665 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:29.077592 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-789f7f47d6-tcbm2" event={"ID":"68724704-7c68-4fdf-9a76-b20b095de703","Type":"ContainerStarted","Data":"7f2e6312455483ba1b4f9343e5cab7735b0ce84c4715f8d8942877418b54db24"} Sep 09 21:00:29.077665 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:29.077637 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-789f7f47d6-tcbm2" event={"ID":"68724704-7c68-4fdf-9a76-b20b095de703","Type":"ContainerStarted","Data":"ed9b5855abda565a57b52a8312b8f78b54b39155c71d8883a80a7ba3274dbe0c"} Sep 09 21:00:29.096003 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:29.095959 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-789f7f47d6-tcbm2" podStartSLOduration=2.095944343 podStartE2EDuration="2.095944343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:29.094279173 +0000 UTC m=+606.554921090" watchObservedRunningTime="2026-09-09 21:00:29.095944343 +0000 UTC m=+606.556586264" Sep 09 21:00:33.384628 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.384596 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5fc8dd9cff-8tm7j"] Sep 09 21:00:33.387926 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.387886 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5fc8dd9cff-8tm7j" Sep 09 21:00:33.391046 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.391028 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 09 21:00:33.391152 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.391029 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 09 21:00:33.391567 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.391552 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 09 21:00:33.391865 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.391847 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 09 21:00:33.392149 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.392130 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 09 21:00:33.421386 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.421367 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad6dd721-249e-499e-b34f-0196649300fa-metrics-cert\") pod \"lws-controller-manager-5fc8dd9cff-8tm7j\" (UID: \"ad6dd721-249e-499e-b34f-0196649300fa\") " pod="openshift-lws-operator/lws-controller-manager-5fc8dd9cff-8tm7j" Sep 09 21:00:33.421477 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.421396 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xtt4\" (UniqueName: \"kubernetes.io/projected/ad6dd721-249e-499e-b34f-0196649300fa-kube-api-access-8xtt4\") pod \"lws-controller-manager-5fc8dd9cff-8tm7j\" (UID: \"ad6dd721-249e-499e-b34f-0196649300fa\") " pod="openshift-lws-operator/lws-controller-manager-5fc8dd9cff-8tm7j" Sep 09 21:00:33.421477 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.421439 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ad6dd721-249e-499e-b34f-0196649300fa-manager-config\") pod \"lws-controller-manager-5fc8dd9cff-8tm7j\" (UID: \"ad6dd721-249e-499e-b34f-0196649300fa\") " pod="openshift-lws-operator/lws-controller-manager-5fc8dd9cff-8tm7j" Sep 09 21:00:33.421546 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.421510 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad6dd721-249e-499e-b34f-0196649300fa-cert\") pod \"lws-controller-manager-5fc8dd9cff-8tm7j\" (UID: \"ad6dd721-249e-499e-b34f-0196649300fa\") " pod="openshift-lws-operator/lws-controller-manager-5fc8dd9cff-8tm7j" Sep 09 21:00:33.426007 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.425992 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8txq6\"" Sep 09 21:00:33.427177 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.427160 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5fc8dd9cff-8tm7j"] Sep 09 21:00:33.522195 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.522171 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ad6dd721-249e-499e-b34f-0196649300fa-manager-config\") pod \"lws-controller-manager-5fc8dd9cff-8tm7j\" (UID: \"ad6dd721-249e-499e-b34f-0196649300fa\") " pod="openshift-lws-operator/lws-controller-manager-5fc8dd9cff-8tm7j" Sep 09 21:00:33.522291 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.522207 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad6dd721-249e-499e-b34f-0196649300fa-cert\") pod \"lws-controller-manager-5fc8dd9cff-8tm7j\" (UID: \"ad6dd721-249e-499e-b34f-0196649300fa\") " pod="openshift-lws-operator/lws-controller-manager-5fc8dd9cff-8tm7j" Sep 09 21:00:33.522291 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.522234 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad6dd721-249e-499e-b34f-0196649300fa-metrics-cert\") pod \"lws-controller-manager-5fc8dd9cff-8tm7j\" (UID: \"ad6dd721-249e-499e-b34f-0196649300fa\") " pod="openshift-lws-operator/lws-controller-manager-5fc8dd9cff-8tm7j" Sep 09 21:00:33.522291 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.522253 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xtt4\" (UniqueName: \"kubernetes.io/projected/ad6dd721-249e-499e-b34f-0196649300fa-kube-api-access-8xtt4\") pod \"lws-controller-manager-5fc8dd9cff-8tm7j\" (UID: \"ad6dd721-249e-499e-b34f-0196649300fa\") " pod="openshift-lws-operator/lws-controller-manager-5fc8dd9cff-8tm7j" Sep 09 21:00:33.522779 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.522753 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ad6dd721-249e-499e-b34f-0196649300fa-manager-config\") pod \"lws-controller-manager-5fc8dd9cff-8tm7j\" (UID: \"ad6dd721-249e-499e-b34f-0196649300fa\") " pod="openshift-lws-operator/lws-controller-manager-5fc8dd9cff-8tm7j" Sep 09 21:00:33.524792 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.524766 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad6dd721-249e-499e-b34f-0196649300fa-metrics-cert\") pod \"lws-controller-manager-5fc8dd9cff-8tm7j\" (UID: \"ad6dd721-249e-499e-b34f-0196649300fa\") " pod="openshift-lws-operator/lws-controller-manager-5fc8dd9cff-8tm7j" Sep 09 21:00:33.524867 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.524818 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad6dd721-249e-499e-b34f-0196649300fa-cert\") pod \"lws-controller-manager-5fc8dd9cff-8tm7j\" (UID: \"ad6dd721-249e-499e-b34f-0196649300fa\") " pod="openshift-lws-operator/lws-controller-manager-5fc8dd9cff-8tm7j" Sep 09 21:00:33.562200 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.562179 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xtt4\" (UniqueName: \"kubernetes.io/projected/ad6dd721-249e-499e-b34f-0196649300fa-kube-api-access-8xtt4\") pod \"lws-controller-manager-5fc8dd9cff-8tm7j\" (UID: \"ad6dd721-249e-499e-b34f-0196649300fa\") " pod="openshift-lws-operator/lws-controller-manager-5fc8dd9cff-8tm7j" Sep 09 21:00:33.696951 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.696927 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5fc8dd9cff-8tm7j" Sep 09 21:00:33.717822 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.717792 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:33.817831 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.817806 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-m6bbc"] Sep 09 21:00:33.821164 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.821146 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m6bbc" Sep 09 21:00:33.826790 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.826769 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 09 21:00:33.827955 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.827933 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-42pp4\"" Sep 09 21:00:33.833164 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.833146 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-njk78"] Sep 09 21:00:33.835927 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.835881 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njk78" Sep 09 21:00:33.839207 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.839177 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 09 21:00:33.839318 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.839303 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-m9fl8\"" Sep 09 21:00:33.842486 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.842467 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-m6bbc"] Sep 09 21:00:33.863557 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.863536 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-njk78"] Sep 09 21:00:33.919237 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.919199 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:33.926152 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.926133 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64b5633f-b9c9-4d46-9224-fd46d2e5cb01-cert\") pod \"kserve-controller-manager-d6f4b65b8-m6bbc\" (UID: \"64b5633f-b9c9-4d46-9224-fd46d2e5cb01\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m6bbc" Sep 09 21:00:33.926249 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.926192 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7wbc\" (UniqueName: \"kubernetes.io/projected/64b5633f-b9c9-4d46-9224-fd46d2e5cb01-kube-api-access-m7wbc\") pod \"kserve-controller-manager-d6f4b65b8-m6bbc\" (UID: \"64b5633f-b9c9-4d46-9224-fd46d2e5cb01\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m6bbc" Sep 09 21:00:33.926308 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.926247 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5tqt\" (UniqueName: \"kubernetes.io/projected/4e582a30-e52b-4a4e-9b26-f4e0f7ec57d4-kube-api-access-s5tqt\") pod \"llmisvc-controller-manager-777c6bd7d5-njk78\" (UID: \"4e582a30-e52b-4a4e-9b26-f4e0f7ec57d4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njk78" Sep 09 21:00:33.926308 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.926283 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e582a30-e52b-4a4e-9b26-f4e0f7ec57d4-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-njk78\" (UID: \"4e582a30-e52b-4a4e-9b26-f4e0f7ec57d4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njk78" Sep 09 21:00:33.979228 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.979210 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5fc8dd9cff-8tm7j"] Sep 09 21:00:33.981469 ip-10-0-137-19 kubenswrapper[2580]: W0909 21:00:33.981449 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad6dd721_249e_499e_b34f_0196649300fa.slice/crio-de449c3c4b6a90721591d5cafd1a135798b1c5b43721489435a37a6cd126ffd4 WatchSource:0}: Error finding container de449c3c4b6a90721591d5cafd1a135798b1c5b43721489435a37a6cd126ffd4: Status 404 returned error can't find the container with id de449c3c4b6a90721591d5cafd1a135798b1c5b43721489435a37a6cd126ffd4 Sep 09 21:00:33.983273 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:33.983261 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 21:00:34.026617 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:34.026598 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7wbc\" (UniqueName: \"kubernetes.io/projected/64b5633f-b9c9-4d46-9224-fd46d2e5cb01-kube-api-access-m7wbc\") pod \"kserve-controller-manager-d6f4b65b8-m6bbc\" (UID: \"64b5633f-b9c9-4d46-9224-fd46d2e5cb01\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m6bbc" Sep 09 21:00:34.026702 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:34.026627 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5tqt\" (UniqueName: \"kubernetes.io/projected/4e582a30-e52b-4a4e-9b26-f4e0f7ec57d4-kube-api-access-s5tqt\") pod \"llmisvc-controller-manager-777c6bd7d5-njk78\" (UID: \"4e582a30-e52b-4a4e-9b26-f4e0f7ec57d4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njk78" Sep 09 21:00:34.026702 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:34.026654 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e582a30-e52b-4a4e-9b26-f4e0f7ec57d4-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-njk78\" (UID: \"4e582a30-e52b-4a4e-9b26-f4e0f7ec57d4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njk78" Sep 09 21:00:34.026790 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:34.026700 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64b5633f-b9c9-4d46-9224-fd46d2e5cb01-cert\") pod \"kserve-controller-manager-d6f4b65b8-m6bbc\" (UID: \"64b5633f-b9c9-4d46-9224-fd46d2e5cb01\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m6bbc" Sep 09 21:00:34.026829 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:00:34.026807 2580 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 09 21:00:34.026829 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:00:34.026811 2580 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 09 21:00:34.026912 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:00:34.026853 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64b5633f-b9c9-4d46-9224-fd46d2e5cb01-cert podName:64b5633f-b9c9-4d46-9224-fd46d2e5cb01 nodeName:}" failed. No retries permitted until 2026-09-09 21:00:34.526840529 +0000 UTC m=+611.987482429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/64b5633f-b9c9-4d46-9224-fd46d2e5cb01-cert") pod "kserve-controller-manager-d6f4b65b8-m6bbc" (UID: "64b5633f-b9c9-4d46-9224-fd46d2e5cb01") : secret "kserve-webhook-server-cert" not found Sep 09 21:00:34.026912 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:00:34.026864 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e582a30-e52b-4a4e-9b26-f4e0f7ec57d4-cert podName:4e582a30-e52b-4a4e-9b26-f4e0f7ec57d4 nodeName:}" failed. No retries permitted until 2026-09-09 21:00:34.526858553 +0000 UTC m=+611.987500453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4e582a30-e52b-4a4e-9b26-f4e0f7ec57d4-cert") pod "llmisvc-controller-manager-777c6bd7d5-njk78" (UID: "4e582a30-e52b-4a4e-9b26-f4e0f7ec57d4") : secret "llmisvc-webhook-server-cert" not found Sep 09 21:00:34.045201 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:34.045176 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5tqt\" (UniqueName: \"kubernetes.io/projected/4e582a30-e52b-4a4e-9b26-f4e0f7ec57d4-kube-api-access-s5tqt\") pod \"llmisvc-controller-manager-777c6bd7d5-njk78\" (UID: \"4e582a30-e52b-4a4e-9b26-f4e0f7ec57d4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njk78" Sep 09 21:00:34.048819 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:34.048795 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7wbc\" (UniqueName: \"kubernetes.io/projected/64b5633f-b9c9-4d46-9224-fd46d2e5cb01-kube-api-access-m7wbc\") pod \"kserve-controller-manager-d6f4b65b8-m6bbc\" (UID: \"64b5633f-b9c9-4d46-9224-fd46d2e5cb01\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m6bbc" Sep 09 21:00:34.096954 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:34.096925 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5fc8dd9cff-8tm7j" event={"ID":"ad6dd721-249e-499e-b34f-0196649300fa","Type":"ContainerStarted","Data":"de449c3c4b6a90721591d5cafd1a135798b1c5b43721489435a37a6cd126ffd4"} Sep 09 21:00:34.115424 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:34.115399 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:34.316552 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:34.316501 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:34.530704 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:34.530650 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64b5633f-b9c9-4d46-9224-fd46d2e5cb01-cert\") pod \"kserve-controller-manager-d6f4b65b8-m6bbc\" (UID: \"64b5633f-b9c9-4d46-9224-fd46d2e5cb01\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m6bbc" Sep 09 21:00:34.531135 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:34.530752 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e582a30-e52b-4a4e-9b26-f4e0f7ec57d4-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-njk78\" (UID: \"4e582a30-e52b-4a4e-9b26-f4e0f7ec57d4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njk78" Sep 09 21:00:34.533339 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:34.533310 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e582a30-e52b-4a4e-9b26-f4e0f7ec57d4-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-njk78\" (UID: \"4e582a30-e52b-4a4e-9b26-f4e0f7ec57d4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njk78" Sep 09 21:00:34.533439 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:34.533381 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64b5633f-b9c9-4d46-9224-fd46d2e5cb01-cert\") pod \"kserve-controller-manager-d6f4b65b8-m6bbc\" (UID: \"64b5633f-b9c9-4d46-9224-fd46d2e5cb01\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-m6bbc" Sep 09 21:00:34.731035 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:34.731010 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m6bbc" Sep 09 21:00:34.744837 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:34.744810 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njk78" Sep 09 21:00:35.031132 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:35.031086 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-m6bbc"] Sep 09 21:00:35.035083 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:35.035057 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-njk78"] Sep 09 21:00:35.035623 ip-10-0-137-19 kubenswrapper[2580]: W0909 21:00:35.035596 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b5633f_b9c9_4d46_9224_fd46d2e5cb01.slice/crio-fadcb20e2bbf98fe41e8e5d46441a9270d7eee509435d9b8bf83f6772ad125fc WatchSource:0}: Error finding container fadcb20e2bbf98fe41e8e5d46441a9270d7eee509435d9b8bf83f6772ad125fc: Status 404 returned error can't find the container with id fadcb20e2bbf98fe41e8e5d46441a9270d7eee509435d9b8bf83f6772ad125fc Sep 09 21:00:35.037492 ip-10-0-137-19 kubenswrapper[2580]: W0909 21:00:35.037453 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4e582a30_e52b_4a4e_9b26_f4e0f7ec57d4.slice/crio-5bdd5f27d3f5b705540a08e1423c1a320fb06b0669dc372004987a0e84db6d2d WatchSource:0}: Error finding container 5bdd5f27d3f5b705540a08e1423c1a320fb06b0669dc372004987a0e84db6d2d: Status 404 returned error can't find the container with id 5bdd5f27d3f5b705540a08e1423c1a320fb06b0669dc372004987a0e84db6d2d Sep 09 21:00:35.103512 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:35.103166 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njk78" event={"ID":"4e582a30-e52b-4a4e-9b26-f4e0f7ec57d4","Type":"ContainerStarted","Data":"5bdd5f27d3f5b705540a08e1423c1a320fb06b0669dc372004987a0e84db6d2d"} Sep 09 21:00:35.104818 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:35.104788 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m6bbc" event={"ID":"64b5633f-b9c9-4d46-9224-fd46d2e5cb01","Type":"ContainerStarted","Data":"fadcb20e2bbf98fe41e8e5d46441a9270d7eee509435d9b8bf83f6772ad125fc"} Sep 09 21:00:35.109059 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:35.109034 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:35.213011 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:35.212984 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:38.134462 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:38.134432 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:38.134807 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:38.134484 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:38.139875 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:38.139854 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:39.124454 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:39.124364 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m6bbc" event={"ID":"64b5633f-b9c9-4d46-9224-fd46d2e5cb01","Type":"ContainerStarted","Data":"1f53373e11c0188be6ff788e0f7434dadd989010647f15f0a283002dafd55cb7"} Sep 09 21:00:39.124622 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:39.124474 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m6bbc" Sep 09 21:00:39.126380 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:39.126352 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5fc8dd9cff-8tm7j" event={"ID":"ad6dd721-249e-499e-b34f-0196649300fa","Type":"ContainerStarted","Data":"7fc2b043d82c630579f9f24f0e3990521c5124a9a151c9f0d055485f9992e9ba"} Sep 09 21:00:39.126525 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:39.126459 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5fc8dd9cff-8tm7j" Sep 09 21:00:39.130114 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:39.130096 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-789f7f47d6-tcbm2" Sep 09 21:00:39.141779 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:39.141734 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m6bbc" podStartSLOduration=2.694530413 podStartE2EDuration="6.141722003s" totalImagesPullingTime="3.44719159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:39.140055047 +0000 UTC m=+616.600696970" watchObservedRunningTime="2026-09-09 21:00:39.141722003 +0000 UTC m=+616.602363941" Sep 09 21:00:39.174883 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:39.174839 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5fc8dd9cff-8tm7j" podStartSLOduration=1.993037422 podStartE2EDuration="6.174826876s" totalImagesPullingTime="4.181789454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:39.17441968 +0000 UTC m=+616.635061614" watchObservedRunningTime="2026-09-09 21:00:39.174826876 +0000 UTC m=+616.635468798" Sep 09 21:00:39.190802 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:39.190778 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8c7bcd8d4-vzfdp"] Sep 09 21:00:39.363460 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:39.363435 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:39.509987 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:39.509956 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:39.614591 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:39.614552 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:41.136116 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:41.136084 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njk78" event={"ID":"4e582a30-e52b-4a4e-9b26-f4e0f7ec57d4","Type":"ContainerStarted","Data":"c8559c085733a802e87d70ba5c6aa89c4ac3778001eff2fef9a0cb927ea5f7eb"} Sep 09 21:00:41.136461 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:41.136185 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njk78" Sep 09 21:00:41.158203 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:41.158159 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njk78" podStartSLOduration=2.742962298 podStartE2EDuration="8.158146128s" totalImagesPullingTime="5.41518383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:41.156466649 +0000 UTC m=+618.617108572" watchObservedRunningTime="2026-09-09 21:00:41.158146128 +0000 UTC m=+618.618788050" Sep 09 21:00:50.132847 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:00:50.132776 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5fc8dd9cff-8tm7j" Sep 09 21:01:04.210500 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:04.210456 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8c7bcd8d4-vzfdp" podUID="72bb276c-bd16-47d6-a429-58a41d4302bb" containerName="console" containerID="cri-o://2c5c1076e7028ca3f94f97e4b6dbf1b2bcc614321882d283b01df85b510ad877" gracePeriod=15 Sep 09 21:01:04.443582 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:04.443561 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 21:01:04.552202 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:04.552129 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72bb276c-bd16-47d6-a429-58a41d4302bb-trusted-ca-bundle\") pod \"72bb276c-bd16-47d6-a429-58a41d4302bb\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " Sep 09 21:01:04.552202 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:04.552164 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72bb276c-bd16-47d6-a429-58a41d4302bb-service-ca\") pod \"72bb276c-bd16-47d6-a429-58a41d4302bb\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " Sep 09 21:01:04.552410 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:04.552205 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72bb276c-bd16-47d6-a429-58a41d4302bb-console-oauth-config\") pod \"72bb276c-bd16-47d6-a429-58a41d4302bb\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " Sep 09 21:01:04.552410 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:04.552240 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72bb276c-bd16-47d6-a429-58a41d4302bb-oauth-serving-cert\") pod \"72bb276c-bd16-47d6-a429-58a41d4302bb\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " Sep 09 21:01:04.552410 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:04.552270 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72bb276c-bd16-47d6-a429-58a41d4302bb-console-config\") pod \"72bb276c-bd16-47d6-a429-58a41d4302bb\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " Sep 09 21:01:04.552410 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:04.552284 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72bb276c-bd16-47d6-a429-58a41d4302bb-console-serving-cert\") pod \"72bb276c-bd16-47d6-a429-58a41d4302bb\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " Sep 09 21:01:04.552410 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:04.552315 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jz954\" (UniqueName: \"kubernetes.io/projected/72bb276c-bd16-47d6-a429-58a41d4302bb-kube-api-access-jz954\") pod \"72bb276c-bd16-47d6-a429-58a41d4302bb\" (UID: \"72bb276c-bd16-47d6-a429-58a41d4302bb\") " Sep 09 21:01:04.552650 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:04.552582 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72bb276c-bd16-47d6-a429-58a41d4302bb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "72bb276c-bd16-47d6-a429-58a41d4302bb" (UID: "72bb276c-bd16-47d6-a429-58a41d4302bb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 21:01:04.552737 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:04.552713 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72bb276c-bd16-47d6-a429-58a41d4302bb-service-ca" (OuterVolumeSpecName: "service-ca") pod "72bb276c-bd16-47d6-a429-58a41d4302bb" (UID: "72bb276c-bd16-47d6-a429-58a41d4302bb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 21:01:04.552737 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:04.552718 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72bb276c-bd16-47d6-a429-58a41d4302bb-console-config" (OuterVolumeSpecName: "console-config") pod "72bb276c-bd16-47d6-a429-58a41d4302bb" (UID: "72bb276c-bd16-47d6-a429-58a41d4302bb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 21:01:04.552832 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:04.552729 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72bb276c-bd16-47d6-a429-58a41d4302bb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "72bb276c-bd16-47d6-a429-58a41d4302bb" (UID: "72bb276c-bd16-47d6-a429-58a41d4302bb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 21:01:04.554418 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:04.554390 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72bb276c-bd16-47d6-a429-58a41d4302bb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "72bb276c-bd16-47d6-a429-58a41d4302bb" (UID: "72bb276c-bd16-47d6-a429-58a41d4302bb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 21:01:04.554512 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:04.554452 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72bb276c-bd16-47d6-a429-58a41d4302bb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "72bb276c-bd16-47d6-a429-58a41d4302bb" (UID: "72bb276c-bd16-47d6-a429-58a41d4302bb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 21:01:04.554557 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:04.554539 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72bb276c-bd16-47d6-a429-58a41d4302bb-kube-api-access-jz954" (OuterVolumeSpecName: "kube-api-access-jz954") pod "72bb276c-bd16-47d6-a429-58a41d4302bb" (UID: "72bb276c-bd16-47d6-a429-58a41d4302bb"). InnerVolumeSpecName "kube-api-access-jz954". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 21:01:04.653526 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:04.653502 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72bb276c-bd16-47d6-a429-58a41d4302bb-trusted-ca-bundle\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 21:01:04.653526 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:04.653521 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72bb276c-bd16-47d6-a429-58a41d4302bb-service-ca\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 21:01:04.653647 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:04.653531 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72bb276c-bd16-47d6-a429-58a41d4302bb-console-oauth-config\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 21:01:04.653647 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:04.653539 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72bb276c-bd16-47d6-a429-58a41d4302bb-oauth-serving-cert\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 21:01:04.653647 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:04.653548 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72bb276c-bd16-47d6-a429-58a41d4302bb-console-config\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 21:01:04.653647 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:04.653556 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72bb276c-bd16-47d6-a429-58a41d4302bb-console-serving-cert\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 21:01:04.653647 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:04.653563 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz954\" (UniqueName: \"kubernetes.io/projected/72bb276c-bd16-47d6-a429-58a41d4302bb-kube-api-access-jz954\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 21:01:05.221992 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:05.221968 2580 generic.go:356] "Generic (PLEG): container finished" podID="72bb276c-bd16-47d6-a429-58a41d4302bb" containerID="2c5c1076e7028ca3f94f97e4b6dbf1b2bcc614321882d283b01df85b510ad877" exitCode=0 Sep 09 21:01:05.222303 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:05.222013 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8c7bcd8d4-vzfdp" Sep 09 21:01:05.222303 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:05.222056 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8c7bcd8d4-vzfdp" event={"ID":"72bb276c-bd16-47d6-a429-58a41d4302bb","Type":"ContainerDied","Data":"2c5c1076e7028ca3f94f97e4b6dbf1b2bcc614321882d283b01df85b510ad877"} Sep 09 21:01:05.222303 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:05.222104 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8c7bcd8d4-vzfdp" event={"ID":"72bb276c-bd16-47d6-a429-58a41d4302bb","Type":"ContainerDied","Data":"b398be1082f2d95e74f8be1d16b20a0aa205beec31030859c93dcb1693946a2c"} Sep 09 21:01:05.222303 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:05.222129 2580 scope.go:117] "RemoveContainer" containerID="2c5c1076e7028ca3f94f97e4b6dbf1b2bcc614321882d283b01df85b510ad877" Sep 09 21:01:05.230735 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:05.230719 2580 scope.go:117] "RemoveContainer" containerID="2c5c1076e7028ca3f94f97e4b6dbf1b2bcc614321882d283b01df85b510ad877" Sep 09 21:01:05.231036 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:01:05.231015 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c5c1076e7028ca3f94f97e4b6dbf1b2bcc614321882d283b01df85b510ad877\": container with ID starting with 2c5c1076e7028ca3f94f97e4b6dbf1b2bcc614321882d283b01df85b510ad877 not found: ID does not exist" containerID="2c5c1076e7028ca3f94f97e4b6dbf1b2bcc614321882d283b01df85b510ad877" Sep 09 21:01:05.231089 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:05.231046 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c5c1076e7028ca3f94f97e4b6dbf1b2bcc614321882d283b01df85b510ad877"} err="failed to get container status \"2c5c1076e7028ca3f94f97e4b6dbf1b2bcc614321882d283b01df85b510ad877\": rpc error: code = NotFound desc = could not find container \"2c5c1076e7028ca3f94f97e4b6dbf1b2bcc614321882d283b01df85b510ad877\": container with ID starting with 2c5c1076e7028ca3f94f97e4b6dbf1b2bcc614321882d283b01df85b510ad877 not found: ID does not exist" Sep 09 21:01:05.238369 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:05.238347 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8c7bcd8d4-vzfdp"] Sep 09 21:01:05.242124 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:05.242104 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8c7bcd8d4-vzfdp"] Sep 09 21:01:07.042532 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:07.042488 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72bb276c-bd16-47d6-a429-58a41d4302bb" path="/var/lib/kubelet/pods/72bb276c-bd16-47d6-a429-58a41d4302bb/volumes" Sep 09 21:01:10.135793 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:10.135768 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-m6bbc" Sep 09 21:01:12.142331 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:12.142303 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njk78" Sep 09 21:01:25.285058 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:25.285024 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:25.375808 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:25.375781 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:39.095219 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:39.095179 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:39.885541 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:39.885484 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:40.711991 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:40.711949 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:42.473769 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:42.473732 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:42.573327 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:42.573298 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:43.805668 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:43.805627 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:43.902955 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:43.902921 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:44.366913 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:44.366852 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:45.777627 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:45.777597 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h9s2p"] Sep 09 21:01:45.778061 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:45.777852 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72bb276c-bd16-47d6-a429-58a41d4302bb" containerName="console" Sep 09 21:01:45.778061 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:45.777863 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="72bb276c-bd16-47d6-a429-58a41d4302bb" containerName="console" Sep 09 21:01:45.778061 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:45.777933 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="72bb276c-bd16-47d6-a429-58a41d4302bb" containerName="console" Sep 09 21:01:45.781946 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:45.781930 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h9s2p" Sep 09 21:01:45.786032 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:45.786010 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-tzv6h\"" Sep 09 21:01:45.786161 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:45.786069 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 09 21:01:45.786161 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:45.786068 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 09 21:01:45.786161 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:45.786072 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 09 21:01:45.809906 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:45.809864 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h9s2p"] Sep 09 21:01:45.830714 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:45.830690 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4dld\" (UniqueName: \"kubernetes.io/projected/1936b418-b1a9-4731-9fe1-b63206b10982-kube-api-access-j4dld\") pod \"dns-operator-controller-manager-7b84cd7c9c-h9s2p\" (UID: \"1936b418-b1a9-4731-9fe1-b63206b10982\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h9s2p" Sep 09 21:01:45.931984 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:45.931954 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4dld\" (UniqueName: \"kubernetes.io/projected/1936b418-b1a9-4731-9fe1-b63206b10982-kube-api-access-j4dld\") pod \"dns-operator-controller-manager-7b84cd7c9c-h9s2p\" (UID: \"1936b418-b1a9-4731-9fe1-b63206b10982\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h9s2p" Sep 09 21:01:45.951363 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:45.951338 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4dld\" (UniqueName: \"kubernetes.io/projected/1936b418-b1a9-4731-9fe1-b63206b10982-kube-api-access-j4dld\") pod \"dns-operator-controller-manager-7b84cd7c9c-h9s2p\" (UID: \"1936b418-b1a9-4731-9fe1-b63206b10982\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h9s2p" Sep 09 21:01:45.980321 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:45.980298 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:46.091822 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:46.091765 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h9s2p" Sep 09 21:01:46.327827 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:46.327788 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:46.397094 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:46.394750 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h9s2p"] Sep 09 21:01:46.400920 ip-10-0-137-19 kubenswrapper[2580]: W0909 21:01:46.400868 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1936b418_b1a9_4731_9fe1_b63206b10982.slice/crio-f7b904778ea9de46b98fdfa94b51037ea46fcf46df63c1910b8b36fa1e7a0d66 WatchSource:0}: Error finding container f7b904778ea9de46b98fdfa94b51037ea46fcf46df63c1910b8b36fa1e7a0d66: Status 404 returned error can't find the container with id f7b904778ea9de46b98fdfa94b51037ea46fcf46df63c1910b8b36fa1e7a0d66 Sep 09 21:01:46.486646 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:46.486617 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:47.382114 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:47.382070 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h9s2p" event={"ID":"1936b418-b1a9-4731-9fe1-b63206b10982","Type":"ContainerStarted","Data":"f7b904778ea9de46b98fdfa94b51037ea46fcf46df63c1910b8b36fa1e7a0d66"} Sep 09 21:01:48.693601 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:48.693565 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-p5fdj"] Sep 09 21:01:48.697442 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:48.697415 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-p5fdj" Sep 09 21:01:48.699777 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:48.699757 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-v64cr\"" Sep 09 21:01:48.713009 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:48.712986 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-p5fdj"] Sep 09 21:01:48.756602 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:48.756579 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4p7j\" (UniqueName: \"kubernetes.io/projected/96951f8a-31a7-4b58-ae9d-7a72a1333071-kube-api-access-w4p7j\") pod \"authorino-operator-76b478bd78-p5fdj\" (UID: \"96951f8a-31a7-4b58-ae9d-7a72a1333071\") " pod="kuadrant-system/authorino-operator-76b478bd78-p5fdj" Sep 09 21:01:48.857321 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:48.857297 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4p7j\" (UniqueName: \"kubernetes.io/projected/96951f8a-31a7-4b58-ae9d-7a72a1333071-kube-api-access-w4p7j\") pod \"authorino-operator-76b478bd78-p5fdj\" (UID: \"96951f8a-31a7-4b58-ae9d-7a72a1333071\") " pod="kuadrant-system/authorino-operator-76b478bd78-p5fdj" Sep 09 21:01:48.870338 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:48.870311 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4p7j\" (UniqueName: \"kubernetes.io/projected/96951f8a-31a7-4b58-ae9d-7a72a1333071-kube-api-access-w4p7j\") pod \"authorino-operator-76b478bd78-p5fdj\" (UID: \"96951f8a-31a7-4b58-ae9d-7a72a1333071\") " pod="kuadrant-system/authorino-operator-76b478bd78-p5fdj" Sep 09 21:01:49.008008 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:49.007952 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-p5fdj" Sep 09 21:01:49.033111 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:49.033085 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:49.186026 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:49.185992 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:49.293261 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:49.293238 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-p5fdj"] Sep 09 21:01:49.295721 ip-10-0-137-19 kubenswrapper[2580]: W0909 21:01:49.295697 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96951f8a_31a7_4b58_ae9d_7a72a1333071.slice/crio-1a4d429a943fa829b7a8ebb93562bdb3a81888e741302a8e73af2a283688920c WatchSource:0}: Error finding container 1a4d429a943fa829b7a8ebb93562bdb3a81888e741302a8e73af2a283688920c: Status 404 returned error can't find the container with id 1a4d429a943fa829b7a8ebb93562bdb3a81888e741302a8e73af2a283688920c Sep 09 21:01:49.391425 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:49.391399 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h9s2p" event={"ID":"1936b418-b1a9-4731-9fe1-b63206b10982","Type":"ContainerStarted","Data":"06b2cb63ce5c37fd29a7f826e957b17b34201a85b4e15790cb8b81cc2c8aa3a2"} Sep 09 21:01:49.391542 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:49.391528 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h9s2p" Sep 09 21:01:49.392632 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:49.392614 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-p5fdj" event={"ID":"96951f8a-31a7-4b58-ae9d-7a72a1333071","Type":"ContainerStarted","Data":"1a4d429a943fa829b7a8ebb93562bdb3a81888e741302a8e73af2a283688920c"} Sep 09 21:01:49.419622 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:49.419581 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h9s2p" podStartSLOduration=2.048687188 podStartE2EDuration="4.419567761s" totalImagesPullingTime="2.370880573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:49.418432317 +0000 UTC m=+686.879074237" watchObservedRunningTime="2026-09-09 21:01:49.419567761 +0000 UTC m=+686.880209681" Sep 09 21:01:50.808099 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:50.808072 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:51.405025 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:51.404981 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-p5fdj" event={"ID":"96951f8a-31a7-4b58-ae9d-7a72a1333071","Type":"ContainerStarted","Data":"ace26a806340a11863b074903eb9f974a502b808b60392b265aaf9cfc585b476"} Sep 09 21:01:51.405205 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:51.405086 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-p5fdj" Sep 09 21:01:51.436323 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:51.436277 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-p5fdj" podStartSLOduration=2.135073085 podStartE2EDuration="3.436259847s" totalImagesPullingTime="1.301186762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:51.435689736 +0000 UTC m=+688.896331659" watchObservedRunningTime="2026-09-09 21:01:51.436259847 +0000 UTC m=+688.896901769" Sep 09 21:01:51.455667 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:51.455643 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:51.896933 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:51.896905 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:54.418051 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:54.418022 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:54.518484 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:54.518462 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:54.607154 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:54.607121 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:54.924754 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:54.924726 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:55.024000 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:55.023978 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:55.305745 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:55.305721 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:55.373760 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:01:55.373732 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:00.400589 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:00.400560 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-h9s2p" Sep 09 21:02:02.113771 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:02.113732 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:02.206613 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:02.206582 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:02.411518 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:02.411453 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-p5fdj" Sep 09 21:02:04.477375 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:04.477335 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:04.600984 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:04.600953 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:08.324680 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:08.324650 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:08.414849 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:08.414812 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:08.906813 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:08.906783 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:09.008490 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:09.008467 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:12.684161 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:12.684127 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:12.786293 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:12.786265 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:18.717652 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:18.717571 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:18.821223 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:18.821191 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:19.290952 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:19.290921 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:19.351211 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:19.351189 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:19.890601 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:19.890575 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:19.990384 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:19.990344 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:25.892299 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:25.892259 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:25.992625 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:25.992603 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:31.925627 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:31.925597 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:32.025190 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:32.025166 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:33.734741 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.734711 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9"] Sep 09 21:02:33.738060 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.738040 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.740555 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.740529 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 09 21:02:33.740555 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.740529 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 09 21:02:33.740755 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.740531 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 09 21:02:33.740755 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.740624 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-56fzl\"" Sep 09 21:02:33.749867 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.749844 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9"] Sep 09 21:02:33.762737 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.762716 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:33.856388 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.856367 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4597f764-9265-44e0-b17b-660f72201d59-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.856505 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.856394 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4597f764-9265-44e0-b17b-660f72201d59-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.856505 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.856418 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4597f764-9265-44e0-b17b-660f72201d59-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.856505 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.856437 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4597f764-9265-44e0-b17b-660f72201d59-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.856623 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.856501 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4597f764-9265-44e0-b17b-660f72201d59-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.856623 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.856523 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4597f764-9265-44e0-b17b-660f72201d59-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.856623 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.856539 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr2ng\" (UniqueName: \"kubernetes.io/projected/4597f764-9265-44e0-b17b-660f72201d59-kube-api-access-jr2ng\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.856623 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.856559 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4597f764-9265-44e0-b17b-660f72201d59-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.856623 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.856580 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4597f764-9265-44e0-b17b-660f72201d59-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.905002 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.904978 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:33.957172 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.957151 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4597f764-9265-44e0-b17b-660f72201d59-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.957250 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.957184 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4597f764-9265-44e0-b17b-660f72201d59-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.957250 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.957199 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4597f764-9265-44e0-b17b-660f72201d59-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.957250 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.957217 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr2ng\" (UniqueName: \"kubernetes.io/projected/4597f764-9265-44e0-b17b-660f72201d59-kube-api-access-jr2ng\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.957507 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.957479 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4597f764-9265-44e0-b17b-660f72201d59-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.957626 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.957527 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4597f764-9265-44e0-b17b-660f72201d59-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.957626 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.957538 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4597f764-9265-44e0-b17b-660f72201d59-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.957626 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.957608 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4597f764-9265-44e0-b17b-660f72201d59-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.957795 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.957616 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4597f764-9265-44e0-b17b-660f72201d59-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.957795 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.957652 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4597f764-9265-44e0-b17b-660f72201d59-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.957795 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.957719 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4597f764-9265-44e0-b17b-660f72201d59-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.957795 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.957782 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4597f764-9265-44e0-b17b-660f72201d59-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.958024 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.957998 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4597f764-9265-44e0-b17b-660f72201d59-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.958288 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.958268 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4597f764-9265-44e0-b17b-660f72201d59-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.959507 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.959479 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4597f764-9265-44e0-b17b-660f72201d59-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.959978 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.959962 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4597f764-9265-44e0-b17b-660f72201d59-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.965585 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.965560 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr2ng\" (UniqueName: \"kubernetes.io/projected/4597f764-9265-44e0-b17b-660f72201d59-kube-api-access-jr2ng\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:33.966210 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:33.966191 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4597f764-9265-44e0-b17b-660f72201d59-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rmpg9\" (UID: \"4597f764-9265-44e0-b17b-660f72201d59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:34.047240 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:34.047192 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:34.367129 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:34.367068 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9"] Sep 09 21:02:34.370084 ip-10-0-137-19 kubenswrapper[2580]: W0909 21:02:34.370058 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4597f764_9265_44e0_b17b_660f72201d59.slice/crio-bc38979babd07067b2d830fed6d1aaf9607d92e4af8f7fd8fe3442ca279988ef WatchSource:0}: Error finding container bc38979babd07067b2d830fed6d1aaf9607d92e4af8f7fd8fe3442ca279988ef: Status 404 returned error can't find the container with id bc38979babd07067b2d830fed6d1aaf9607d92e4af8f7fd8fe3442ca279988ef Sep 09 21:02:34.414212 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:34.414184 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:34.566937 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:34.566897 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" event={"ID":"4597f764-9265-44e0-b17b-660f72201d59","Type":"ContainerStarted","Data":"bc38979babd07067b2d830fed6d1aaf9607d92e4af8f7fd8fe3442ca279988ef"} Sep 09 21:02:35.735821 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:35.735781 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:35.838581 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:35.838547 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:37.169580 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:37.169541 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 09 21:02:37.169864 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:37.169616 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 09 21:02:37.169864 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:37.169643 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 09 21:02:37.586197 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:37.586166 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" event={"ID":"4597f764-9265-44e0-b17b-660f72201d59","Type":"ContainerStarted","Data":"16700669997c5a0c108779ff774d3c5e6c376f01123988e1b91970dcd6fd68df"} Sep 09 21:02:37.614698 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:37.614651 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" podStartSLOduration=1.817601282 podStartE2EDuration="4.614635847s" totalImagesPullingTime="2.797034565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:02:37.61197365 +0000 UTC m=+735.072615570" watchObservedRunningTime="2026-09-09 21:02:37.614635847 +0000 UTC m=+735.075277768" Sep 09 21:02:38.047740 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:38.047715 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:38.052059 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:38.052036 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:38.590171 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:38.590146 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:38.590840 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:38.590825 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:02:38.629779 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:38.629757 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lgv4z"] Sep 09 21:02:38.633343 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:38.633324 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lgv4z" Sep 09 21:02:38.636615 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:38.636594 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 09 21:02:38.636768 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:38.636750 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jgnlf\"" Sep 09 21:02:38.643360 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:38.643303 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:38.646369 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:38.646346 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lgv4z"] Sep 09 21:02:38.699532 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:38.699505 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/27352ccd-1957-4a89-b345-bcba6f88891e-config-file\") pod \"limitador-limitador-8588749977-lgv4z\" (UID: \"27352ccd-1957-4a89-b345-bcba6f88891e\") " pod="kuadrant-system/limitador-limitador-8588749977-lgv4z" Sep 09 21:02:38.699748 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:38.699723 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7rt7\" (UniqueName: \"kubernetes.io/projected/27352ccd-1957-4a89-b345-bcba6f88891e-kube-api-access-x7rt7\") pod \"limitador-limitador-8588749977-lgv4z\" (UID: \"27352ccd-1957-4a89-b345-bcba6f88891e\") " pod="kuadrant-system/limitador-limitador-8588749977-lgv4z" Sep 09 21:02:38.717072 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:38.717049 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lgv4z"] Sep 09 21:02:38.764400 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:38.764373 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:38.801272 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:38.801252 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/27352ccd-1957-4a89-b345-bcba6f88891e-config-file\") pod \"limitador-limitador-8588749977-lgv4z\" (UID: \"27352ccd-1957-4a89-b345-bcba6f88891e\") " pod="kuadrant-system/limitador-limitador-8588749977-lgv4z" Sep 09 21:02:38.801392 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:38.801319 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7rt7\" (UniqueName: \"kubernetes.io/projected/27352ccd-1957-4a89-b345-bcba6f88891e-kube-api-access-x7rt7\") pod \"limitador-limitador-8588749977-lgv4z\" (UID: \"27352ccd-1957-4a89-b345-bcba6f88891e\") " pod="kuadrant-system/limitador-limitador-8588749977-lgv4z" Sep 09 21:02:38.801800 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:38.801785 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/27352ccd-1957-4a89-b345-bcba6f88891e-config-file\") pod \"limitador-limitador-8588749977-lgv4z\" (UID: \"27352ccd-1957-4a89-b345-bcba6f88891e\") " pod="kuadrant-system/limitador-limitador-8588749977-lgv4z" Sep 09 21:02:38.810214 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:38.810196 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7rt7\" (UniqueName: \"kubernetes.io/projected/27352ccd-1957-4a89-b345-bcba6f88891e-kube-api-access-x7rt7\") pod \"limitador-limitador-8588749977-lgv4z\" (UID: \"27352ccd-1957-4a89-b345-bcba6f88891e\") " pod="kuadrant-system/limitador-limitador-8588749977-lgv4z" Sep 09 21:02:38.864710 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:38.864639 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:38.945269 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:38.945247 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lgv4z" Sep 09 21:02:39.216579 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:39.216436 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lgv4z"] Sep 09 21:02:39.219094 ip-10-0-137-19 kubenswrapper[2580]: W0909 21:02:39.219071 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27352ccd_1957_4a89_b345_bcba6f88891e.slice/crio-3f46049d12c495cb129ba75a07d1ce56eb9199b2c7a7fd21015eb3673da30825 WatchSource:0}: Error finding container 3f46049d12c495cb129ba75a07d1ce56eb9199b2c7a7fd21015eb3673da30825: Status 404 returned error can't find the container with id 3f46049d12c495cb129ba75a07d1ce56eb9199b2c7a7fd21015eb3673da30825 Sep 09 21:02:39.491591 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:39.491529 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:39.589649 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:39.589621 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:39.595060 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:39.595037 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lgv4z" event={"ID":"27352ccd-1957-4a89-b345-bcba6f88891e","Type":"ContainerStarted","Data":"3f46049d12c495cb129ba75a07d1ce56eb9199b2c7a7fd21015eb3673da30825"} Sep 09 21:02:46.625222 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:46.625188 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lgv4z" event={"ID":"27352ccd-1957-4a89-b345-bcba6f88891e","Type":"ContainerStarted","Data":"9df00b0c2a8baf555d66c2c41a6d5e637a202c9fcd890f9df36b6da7dcb5baa0"} Sep 09 21:02:46.625599 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:46.625317 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-lgv4z" Sep 09 21:02:46.641385 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:46.641342 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-lgv4z" podStartSLOduration=2.13295658 podStartE2EDuration="8.641328608s" totalImagesPullingTime="6.508372028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:02:46.640180558 +0000 UTC m=+744.100822494" watchObservedRunningTime="2026-09-09 21:02:46.641328608 +0000 UTC m=+744.101970562" Sep 09 21:02:55.202118 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:55.202085 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lgv4z"] Sep 09 21:02:55.202599 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:55.202254 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-lgv4z" podUID="27352ccd-1957-4a89-b345-bcba6f88891e" containerName="limitador" containerID="cri-o://9df00b0c2a8baf555d66c2c41a6d5e637a202c9fcd890f9df36b6da7dcb5baa0" gracePeriod=30 Sep 09 21:02:55.204239 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:55.204216 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-lgv4z" Sep 09 21:02:55.667684 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:55.667657 2580 generic.go:356] "Generic (PLEG): container finished" podID="27352ccd-1957-4a89-b345-bcba6f88891e" containerID="9df00b0c2a8baf555d66c2c41a6d5e637a202c9fcd890f9df36b6da7dcb5baa0" exitCode=0 Sep 09 21:02:55.667779 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:55.667699 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lgv4z" event={"ID":"27352ccd-1957-4a89-b345-bcba6f88891e","Type":"ContainerDied","Data":"9df00b0c2a8baf555d66c2c41a6d5e637a202c9fcd890f9df36b6da7dcb5baa0"} Sep 09 21:02:56.140937 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:56.140917 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lgv4z" Sep 09 21:02:56.221974 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:56.221949 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7rt7\" (UniqueName: \"kubernetes.io/projected/27352ccd-1957-4a89-b345-bcba6f88891e-kube-api-access-x7rt7\") pod \"27352ccd-1957-4a89-b345-bcba6f88891e\" (UID: \"27352ccd-1957-4a89-b345-bcba6f88891e\") " Sep 09 21:02:56.222284 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:56.221985 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/27352ccd-1957-4a89-b345-bcba6f88891e-config-file\") pod \"27352ccd-1957-4a89-b345-bcba6f88891e\" (UID: \"27352ccd-1957-4a89-b345-bcba6f88891e\") " Sep 09 21:02:56.222364 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:56.222344 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27352ccd-1957-4a89-b345-bcba6f88891e-config-file" (OuterVolumeSpecName: "config-file") pod "27352ccd-1957-4a89-b345-bcba6f88891e" (UID: "27352ccd-1957-4a89-b345-bcba6f88891e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 21:02:56.224230 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:56.224211 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27352ccd-1957-4a89-b345-bcba6f88891e-kube-api-access-x7rt7" (OuterVolumeSpecName: "kube-api-access-x7rt7") pod "27352ccd-1957-4a89-b345-bcba6f88891e" (UID: "27352ccd-1957-4a89-b345-bcba6f88891e"). InnerVolumeSpecName "kube-api-access-x7rt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 21:02:56.323352 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:56.323301 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7rt7\" (UniqueName: \"kubernetes.io/projected/27352ccd-1957-4a89-b345-bcba6f88891e-kube-api-access-x7rt7\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 21:02:56.323352 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:56.323323 2580 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/27352ccd-1957-4a89-b345-bcba6f88891e-config-file\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\"" Sep 09 21:02:56.673325 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:56.673248 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lgv4z" Sep 09 21:02:56.673477 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:56.673245 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lgv4z" event={"ID":"27352ccd-1957-4a89-b345-bcba6f88891e","Type":"ContainerDied","Data":"3f46049d12c495cb129ba75a07d1ce56eb9199b2c7a7fd21015eb3673da30825"} Sep 09 21:02:56.673477 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:56.673385 2580 scope.go:117] "RemoveContainer" containerID="9df00b0c2a8baf555d66c2c41a6d5e637a202c9fcd890f9df36b6da7dcb5baa0" Sep 09 21:02:56.693885 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:56.693851 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lgv4z"] Sep 09 21:02:56.700161 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:56.700142 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lgv4z"] Sep 09 21:02:57.042637 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:57.042607 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27352ccd-1957-4a89-b345-bcba6f88891e" path="/var/lib/kubelet/pods/27352ccd-1957-4a89-b345-bcba6f88891e/volumes" Sep 09 21:02:59.805942 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:59.805905 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:59.903273 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:02:59.903241 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:03:00.229282 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:00.229251 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-sb2sp"] Sep 09 21:03:00.229579 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:00.229565 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27352ccd-1957-4a89-b345-bcba6f88891e" containerName="limitador" Sep 09 21:03:00.229621 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:00.229582 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="27352ccd-1957-4a89-b345-bcba6f88891e" containerName="limitador" Sep 09 21:03:00.229655 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:00.229642 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="27352ccd-1957-4a89-b345-bcba6f88891e" containerName="limitador" Sep 09 21:03:00.235204 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:00.235185 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sb2sp" Sep 09 21:03:00.242326 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:00.242305 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-w25fv\"" Sep 09 21:03:00.243584 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:00.243567 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 09 21:03:00.243683 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:00.243643 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 09 21:03:00.256923 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:00.256884 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 21:03:00.258297 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:00.258277 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-sb2sp"] Sep 09 21:03:00.338510 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:00.338488 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:03:00.355997 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:00.355972 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/266d0b42-bcd5-4101-9876-57c67cb86bb5-data\") pod \"postgres-69fb4594cf-sb2sp\" (UID: \"266d0b42-bcd5-4101-9876-57c67cb86bb5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sb2sp" Sep 09 21:03:00.356097 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:00.356015 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swd2x\" (UniqueName: \"kubernetes.io/projected/266d0b42-bcd5-4101-9876-57c67cb86bb5-kube-api-access-swd2x\") pod \"postgres-69fb4594cf-sb2sp\" (UID: \"266d0b42-bcd5-4101-9876-57c67cb86bb5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sb2sp" Sep 09 21:03:00.457298 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:00.457269 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swd2x\" (UniqueName: \"kubernetes.io/projected/266d0b42-bcd5-4101-9876-57c67cb86bb5-kube-api-access-swd2x\") pod \"postgres-69fb4594cf-sb2sp\" (UID: \"266d0b42-bcd5-4101-9876-57c67cb86bb5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sb2sp" Sep 09 21:03:00.457398 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:00.457359 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/266d0b42-bcd5-4101-9876-57c67cb86bb5-data\") pod \"postgres-69fb4594cf-sb2sp\" (UID: \"266d0b42-bcd5-4101-9876-57c67cb86bb5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sb2sp" Sep 09 21:03:00.457663 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:00.457646 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/266d0b42-bcd5-4101-9876-57c67cb86bb5-data\") pod \"postgres-69fb4594cf-sb2sp\" (UID: \"266d0b42-bcd5-4101-9876-57c67cb86bb5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sb2sp" Sep 09 21:03:00.465744 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:00.465723 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swd2x\" (UniqueName: \"kubernetes.io/projected/266d0b42-bcd5-4101-9876-57c67cb86bb5-kube-api-access-swd2x\") pod \"postgres-69fb4594cf-sb2sp\" (UID: \"266d0b42-bcd5-4101-9876-57c67cb86bb5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sb2sp" Sep 09 21:03:00.544195 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:00.544151 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sb2sp" Sep 09 21:03:00.818138 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:00.818074 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-sb2sp"] Sep 09 21:03:00.820996 ip-10-0-137-19 kubenswrapper[2580]: W0909 21:03:00.820969 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266d0b42_bcd5_4101_9876_57c67cb86bb5.slice/crio-f0515a905279d544ae273e03efe66b9e5e47119b72eb42931578cd690be98f31 WatchSource:0}: Error finding container f0515a905279d544ae273e03efe66b9e5e47119b72eb42931578cd690be98f31: Status 404 returned error can't find the container with id f0515a905279d544ae273e03efe66b9e5e47119b72eb42931578cd690be98f31 Sep 09 21:03:01.698677 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:01.698584 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sb2sp" event={"ID":"266d0b42-bcd5-4101-9876-57c67cb86bb5","Type":"ContainerStarted","Data":"f0515a905279d544ae273e03efe66b9e5e47119b72eb42931578cd690be98f31"} Sep 09 21:03:05.719875 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:05.719848 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sb2sp" event={"ID":"266d0b42-bcd5-4101-9876-57c67cb86bb5","Type":"ContainerStarted","Data":"cde0eec26c3832f6e0e8901ed78221114930c312e6486fdb920cc61de5cbf1f8"} Sep 09 21:03:05.720166 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:05.719941 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sb2sp" Sep 09 21:03:05.737048 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:05.737009 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sb2sp" podStartSLOduration=0.98190517 podStartE2EDuration="5.736995488s" totalImagesPullingTime="4.755090318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:03:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:03:05.736133903 +0000 UTC m=+763.196775828" watchObservedRunningTime="2026-09-09 21:03:05.736995488 +0000 UTC m=+763.197637410" Sep 09 21:03:11.752731 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:11.752697 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sb2sp" Sep 09 21:03:12.214441 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:12.214402 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:03:12.320833 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:12.320806 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:03:18.210332 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:18.210300 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:03:19.154086 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:19.154058 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:03:19.231240 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:19.231208 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:03:19.374721 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:19.374693 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:03:19.390246 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:19.390221 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d8ddb97f8-hzjcq"] Sep 09 21:03:19.393374 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:19.393360 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8ddb97f8-hzjcq" Sep 09 21:03:19.396007 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:19.395984 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 09 21:03:19.396141 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:19.396039 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qkvvl\"" Sep 09 21:03:19.396316 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:19.396288 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 09 21:03:19.396633 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:19.396618 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 09 21:03:19.399209 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:19.399188 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14ba4026-2fb2-4334-964f-06cc61450890-metrics-certs\") pod \"maas-controller-7d8ddb97f8-hzjcq\" (UID: \"14ba4026-2fb2-4334-964f-06cc61450890\") " pod="opendatahub/maas-controller-7d8ddb97f8-hzjcq" Sep 09 21:03:19.399286 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:19.399245 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64zlr\" (UniqueName: \"kubernetes.io/projected/14ba4026-2fb2-4334-964f-06cc61450890-kube-api-access-64zlr\") pod \"maas-controller-7d8ddb97f8-hzjcq\" (UID: \"14ba4026-2fb2-4334-964f-06cc61450890\") " pod="opendatahub/maas-controller-7d8ddb97f8-hzjcq" Sep 09 21:03:19.399336 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:19.399289 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14ba4026-2fb2-4334-964f-06cc61450890-webhook-cert\") pod \"maas-controller-7d8ddb97f8-hzjcq\" (UID: \"14ba4026-2fb2-4334-964f-06cc61450890\") " pod="opendatahub/maas-controller-7d8ddb97f8-hzjcq" Sep 09 21:03:19.405830 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:19.405749 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d8ddb97f8-hzjcq"] Sep 09 21:03:19.499569 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:19.499548 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14ba4026-2fb2-4334-964f-06cc61450890-webhook-cert\") pod \"maas-controller-7d8ddb97f8-hzjcq\" (UID: \"14ba4026-2fb2-4334-964f-06cc61450890\") " pod="opendatahub/maas-controller-7d8ddb97f8-hzjcq" Sep 09 21:03:19.499667 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:19.499587 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14ba4026-2fb2-4334-964f-06cc61450890-metrics-certs\") pod \"maas-controller-7d8ddb97f8-hzjcq\" (UID: \"14ba4026-2fb2-4334-964f-06cc61450890\") " pod="opendatahub/maas-controller-7d8ddb97f8-hzjcq" Sep 09 21:03:19.499667 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:19.499628 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64zlr\" (UniqueName: \"kubernetes.io/projected/14ba4026-2fb2-4334-964f-06cc61450890-kube-api-access-64zlr\") pod \"maas-controller-7d8ddb97f8-hzjcq\" (UID: \"14ba4026-2fb2-4334-964f-06cc61450890\") " pod="opendatahub/maas-controller-7d8ddb97f8-hzjcq" Sep 09 21:03:19.499739 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:03:19.499669 2580 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 09 21:03:19.499739 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:03:19.499726 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14ba4026-2fb2-4334-964f-06cc61450890-webhook-cert podName:14ba4026-2fb2-4334-964f-06cc61450890 nodeName:}" failed. No retries permitted until 2026-09-09 21:03:19.99971174 +0000 UTC m=+777.460353640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/14ba4026-2fb2-4334-964f-06cc61450890-webhook-cert") pod "maas-controller-7d8ddb97f8-hzjcq" (UID: "14ba4026-2fb2-4334-964f-06cc61450890") : secret "maas-controller-webhook-cert" not found Sep 09 21:03:19.499803 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:03:19.499747 2580 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Sep 09 21:03:19.499803 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:03:19.499800 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14ba4026-2fb2-4334-964f-06cc61450890-metrics-certs podName:14ba4026-2fb2-4334-964f-06cc61450890 nodeName:}" failed. No retries permitted until 2026-09-09 21:03:19.999785957 +0000 UTC m=+777.460427859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14ba4026-2fb2-4334-964f-06cc61450890-metrics-certs") pod "maas-controller-7d8ddb97f8-hzjcq" (UID: "14ba4026-2fb2-4334-964f-06cc61450890") : secret "maas-controller-metrics-tls" not found Sep 09 21:03:19.508721 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:19.508703 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64zlr\" (UniqueName: \"kubernetes.io/projected/14ba4026-2fb2-4334-964f-06cc61450890-kube-api-access-64zlr\") pod \"maas-controller-7d8ddb97f8-hzjcq\" (UID: \"14ba4026-2fb2-4334-964f-06cc61450890\") " pod="opendatahub/maas-controller-7d8ddb97f8-hzjcq" Sep 09 21:03:19.882098 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:19.882072 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:03:19.985204 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:19.985174 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:03:20.004227 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:20.004207 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14ba4026-2fb2-4334-964f-06cc61450890-webhook-cert\") pod \"maas-controller-7d8ddb97f8-hzjcq\" (UID: \"14ba4026-2fb2-4334-964f-06cc61450890\") " pod="opendatahub/maas-controller-7d8ddb97f8-hzjcq" Sep 09 21:03:20.004311 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:20.004246 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14ba4026-2fb2-4334-964f-06cc61450890-metrics-certs\") pod \"maas-controller-7d8ddb97f8-hzjcq\" (UID: \"14ba4026-2fb2-4334-964f-06cc61450890\") " pod="opendatahub/maas-controller-7d8ddb97f8-hzjcq" Sep 09 21:03:20.006644 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:20.006624 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/14ba4026-2fb2-4334-964f-06cc61450890-webhook-cert\") pod \"maas-controller-7d8ddb97f8-hzjcq\" (UID: \"14ba4026-2fb2-4334-964f-06cc61450890\") " pod="opendatahub/maas-controller-7d8ddb97f8-hzjcq" Sep 09 21:03:20.006716 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:20.006682 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14ba4026-2fb2-4334-964f-06cc61450890-metrics-certs\") pod \"maas-controller-7d8ddb97f8-hzjcq\" (UID: \"14ba4026-2fb2-4334-964f-06cc61450890\") " pod="opendatahub/maas-controller-7d8ddb97f8-hzjcq" Sep 09 21:03:20.303293 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:20.303265 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d8ddb97f8-hzjcq" Sep 09 21:03:20.577775 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:20.577749 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d8ddb97f8-hzjcq"] Sep 09 21:03:20.580162 ip-10-0-137-19 kubenswrapper[2580]: W0909 21:03:20.580135 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ba4026_2fb2_4334_964f_06cc61450890.slice/crio-01845231ffb37b643cb7c732087fbc15659e08c52c555a87f294ed767e1722ba WatchSource:0}: Error finding container 01845231ffb37b643cb7c732087fbc15659e08c52c555a87f294ed767e1722ba: Status 404 returned error can't find the container with id 01845231ffb37b643cb7c732087fbc15659e08c52c555a87f294ed767e1722ba Sep 09 21:03:20.780827 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:20.780797 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8ddb97f8-hzjcq" event={"ID":"14ba4026-2fb2-4334-964f-06cc61450890","Type":"ContainerStarted","Data":"01845231ffb37b643cb7c732087fbc15659e08c52c555a87f294ed767e1722ba"} Sep 09 21:03:21.635837 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:21.635755 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:03:21.785910 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:21.785794 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:03:22.954834 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:22.954812 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 09 21:03:23.122539 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:23.122483 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:03:23.221167 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:23.221084 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:03:23.797704 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:23.797667 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d8ddb97f8-hzjcq" event={"ID":"14ba4026-2fb2-4334-964f-06cc61450890","Type":"ContainerStarted","Data":"da6aa623262125a4e10fc1339130d50b12b13fb31f148ccbecef14d064470f62"} Sep 09 21:03:23.797866 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:23.797752 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d8ddb97f8-hzjcq" Sep 09 21:03:27.621908 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:27.621859 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:03:27.702855 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:27.702823 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:03:34.804759 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:34.804726 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d8ddb97f8-hzjcq" Sep 09 21:03:34.823437 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:34.823392 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7d8ddb97f8-hzjcq" podStartSLOduration=13.452378298 podStartE2EDuration="15.823379319s" totalImagesPullingTime="2.371001021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:03:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:03:23.827650811 +0000 UTC m=+781.288292733" watchObservedRunningTime="2026-09-09 21:03:34.823379319 +0000 UTC m=+792.284021239" Sep 09 21:03:54.803449 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:54.803363 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:03:54.887998 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:54.887963 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:03:56.359231 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:56.359196 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:03:56.457795 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:56.457758 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c576cc74c-8fjgr"] Sep 09 21:03:56.458124 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:56.457993 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:03:56.462010 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:56.461994 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c576cc74c-8fjgr" Sep 09 21:03:56.464155 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:56.464137 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 09 21:03:56.464262 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:56.464135 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 09 21:03:56.464457 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:56.464442 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8xxhv\"" Sep 09 21:03:56.472629 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:56.472608 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c576cc74c-8fjgr"] Sep 09 21:03:56.567381 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:56.567348 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:03:56.592349 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:56.592328 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87934a77-22ad-4012-92ef-c1bada029bd0-openshift-service-ca\") pod \"maas-api-5c576cc74c-8fjgr\" (UID: \"87934a77-22ad-4012-92ef-c1bada029bd0\") " pod="odh-ai-gateway-infra/maas-api-5c576cc74c-8fjgr" Sep 09 21:03:56.592490 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:56.592355 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87934a77-22ad-4012-92ef-c1bada029bd0-maas-api-tls\") pod \"maas-api-5c576cc74c-8fjgr\" (UID: \"87934a77-22ad-4012-92ef-c1bada029bd0\") " pod="odh-ai-gateway-infra/maas-api-5c576cc74c-8fjgr" Sep 09 21:03:56.592490 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:56.592375 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87934a77-22ad-4012-92ef-c1bada029bd0-metrics-certs\") pod \"maas-api-5c576cc74c-8fjgr\" (UID: \"87934a77-22ad-4012-92ef-c1bada029bd0\") " pod="odh-ai-gateway-infra/maas-api-5c576cc74c-8fjgr" Sep 09 21:03:56.592490 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:56.592410 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t25xv\" (UniqueName: \"kubernetes.io/projected/87934a77-22ad-4012-92ef-c1bada029bd0-kube-api-access-t25xv\") pod \"maas-api-5c576cc74c-8fjgr\" (UID: \"87934a77-22ad-4012-92ef-c1bada029bd0\") " pod="odh-ai-gateway-infra/maas-api-5c576cc74c-8fjgr" Sep 09 21:03:56.667586 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:56.667508 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:03:56.693594 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:56.693573 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87934a77-22ad-4012-92ef-c1bada029bd0-openshift-service-ca\") pod \"maas-api-5c576cc74c-8fjgr\" (UID: \"87934a77-22ad-4012-92ef-c1bada029bd0\") " pod="odh-ai-gateway-infra/maas-api-5c576cc74c-8fjgr" Sep 09 21:03:56.693701 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:56.693601 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87934a77-22ad-4012-92ef-c1bada029bd0-maas-api-tls\") pod \"maas-api-5c576cc74c-8fjgr\" (UID: \"87934a77-22ad-4012-92ef-c1bada029bd0\") " pod="odh-ai-gateway-infra/maas-api-5c576cc74c-8fjgr" Sep 09 21:03:56.693701 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:56.693620 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87934a77-22ad-4012-92ef-c1bada029bd0-metrics-certs\") pod \"maas-api-5c576cc74c-8fjgr\" (UID: \"87934a77-22ad-4012-92ef-c1bada029bd0\") " pod="odh-ai-gateway-infra/maas-api-5c576cc74c-8fjgr" Sep 09 21:03:56.693701 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:56.693659 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t25xv\" (UniqueName: \"kubernetes.io/projected/87934a77-22ad-4012-92ef-c1bada029bd0-kube-api-access-t25xv\") pod \"maas-api-5c576cc74c-8fjgr\" (UID: \"87934a77-22ad-4012-92ef-c1bada029bd0\") " pod="odh-ai-gateway-infra/maas-api-5c576cc74c-8fjgr" Sep 09 21:03:56.694215 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:56.694190 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87934a77-22ad-4012-92ef-c1bada029bd0-openshift-service-ca\") pod \"maas-api-5c576cc74c-8fjgr\" (UID: \"87934a77-22ad-4012-92ef-c1bada029bd0\") " pod="odh-ai-gateway-infra/maas-api-5c576cc74c-8fjgr" Sep 09 21:03:56.696224 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:56.696203 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87934a77-22ad-4012-92ef-c1bada029bd0-maas-api-tls\") pod \"maas-api-5c576cc74c-8fjgr\" (UID: \"87934a77-22ad-4012-92ef-c1bada029bd0\") " pod="odh-ai-gateway-infra/maas-api-5c576cc74c-8fjgr" Sep 09 21:03:56.696302 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:56.696242 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87934a77-22ad-4012-92ef-c1bada029bd0-metrics-certs\") pod \"maas-api-5c576cc74c-8fjgr\" (UID: \"87934a77-22ad-4012-92ef-c1bada029bd0\") " pod="odh-ai-gateway-infra/maas-api-5c576cc74c-8fjgr" Sep 09 21:03:56.704335 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:56.704308 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t25xv\" (UniqueName: \"kubernetes.io/projected/87934a77-22ad-4012-92ef-c1bada029bd0-kube-api-access-t25xv\") pod \"maas-api-5c576cc74c-8fjgr\" (UID: \"87934a77-22ad-4012-92ef-c1bada029bd0\") " pod="odh-ai-gateway-infra/maas-api-5c576cc74c-8fjgr" Sep 09 21:03:56.772882 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:56.772854 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c576cc74c-8fjgr" Sep 09 21:03:56.814061 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:56.814027 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:03:56.912613 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:56.912580 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:03:57.052813 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:57.052783 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c576cc74c-8fjgr"] Sep 09 21:03:57.053788 ip-10-0-137-19 kubenswrapper[2580]: W0909 21:03:57.053763 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87934a77_22ad_4012_92ef_c1bada029bd0.slice/crio-170b6b141ca94c2ea711e15bac4f2a622c9fce987e9513bf8df79f6ca6913940 WatchSource:0}: Error finding container 170b6b141ca94c2ea711e15bac4f2a622c9fce987e9513bf8df79f6ca6913940: Status 404 returned error can't find the container with id 170b6b141ca94c2ea711e15bac4f2a622c9fce987e9513bf8df79f6ca6913940 Sep 09 21:03:57.943740 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:57.943697 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c576cc74c-8fjgr" event={"ID":"87934a77-22ad-4012-92ef-c1bada029bd0","Type":"ContainerStarted","Data":"170b6b141ca94c2ea711e15bac4f2a622c9fce987e9513bf8df79f6ca6913940"} Sep 09 21:03:59.954045 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:59.954014 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c576cc74c-8fjgr" event={"ID":"87934a77-22ad-4012-92ef-c1bada029bd0","Type":"ContainerStarted","Data":"aa181c0f74fd01d1a733225e04b27b63995604e293186c4f24459194bb954631"} Sep 09 21:03:59.954474 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:59.954085 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c576cc74c-8fjgr" Sep 09 21:03:59.973981 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:03:59.973937 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c576cc74c-8fjgr" podStartSLOduration=1.73694527 podStartE2EDuration="3.9739222s" totalImagesPullingTime="2.23697693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:03:59.972868104 +0000 UTC m=+817.433510025" watchObservedRunningTime="2026-09-09 21:03:59.9739222 +0000 UTC m=+817.434564121" Sep 09 21:04:05.963098 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:05.963076 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c576cc74c-8fjgr" Sep 09 21:04:05.982392 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:05.982364 2580 generic.go:356] "Generic (PLEG): container finished" podID="4597f764-9265-44e0-b17b-660f72201d59" containerID="16700669997c5a0c108779ff774d3c5e6c376f01123988e1b91970dcd6fd68df" exitCode=0 Sep 09 21:04:05.982509 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:05.982440 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" event={"ID":"4597f764-9265-44e0-b17b-660f72201d59","Type":"ContainerDied","Data":"16700669997c5a0c108779ff774d3c5e6c376f01123988e1b91970dcd6fd68df"} Sep 09 21:04:05.982757 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:05.982744 2580 scope.go:117] "RemoveContainer" containerID="16700669997c5a0c108779ff774d3c5e6c376f01123988e1b91970dcd6fd68df" Sep 09 21:04:05.983624 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:05.983599 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 09 21:04:05.983684 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:05.983663 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 09 21:04:05.983717 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:05.983693 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 09 21:04:06.926588 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:06.926493 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:04:06.988472 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:06.988437 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" event={"ID":"4597f764-9265-44e0-b17b-660f72201d59","Type":"ContainerStarted","Data":"ee267b457f1d126bf0a86f9201104a3524422030b1ab17120e9220b77dd7d838"} Sep 09 21:04:07.016465 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:07.016428 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:04:07.047899 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:07.047859 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:04:07.048597 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:07.048577 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-rmpg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Sep 09 21:04:07.048654 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:07.048621 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" podUID="4597f764-9265-44e0-b17b-660f72201d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Sep 09 21:04:08.047661 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:08.047626 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-rmpg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Sep 09 21:04:08.048034 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:08.047685 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" podUID="4597f764-9265-44e0-b17b-660f72201d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Sep 09 21:04:08.590928 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:08.590871 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:04:09.047984 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:09.047948 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-rmpg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Sep 09 21:04:09.048416 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:09.048031 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" podUID="4597f764-9265-44e0-b17b-660f72201d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Sep 09 21:04:10.047771 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:10.047742 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-rmpg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Sep 09 21:04:10.048008 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:10.047986 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" podUID="4597f764-9265-44e0-b17b-660f72201d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Sep 09 21:04:11.048181 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:11.048149 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-rmpg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Sep 09 21:04:11.048545 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:11.048199 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" podUID="4597f764-9265-44e0-b17b-660f72201d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Sep 09 21:04:12.051215 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:12.051188 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:04:12.052183 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:12.052165 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rmpg9" Sep 09 21:04:19.289771 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.289740 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn"] Sep 09 21:04:19.292931 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.292911 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:19.295467 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.295447 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 09 21:04:19.295557 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.295483 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 09 21:04:19.296016 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.296002 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5ztcr\"" Sep 09 21:04:19.296099 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.296087 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 09 21:04:19.306403 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.306383 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn"] Sep 09 21:04:19.316784 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.316759 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:04:19.404755 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.404721 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5prx\" (UniqueName: \"kubernetes.io/projected/d957765a-bd0b-4764-9abc-23a64ba4d6d3-kube-api-access-x5prx\") pod \"e2e-distinct-2-simulated-kserve-859856b874-78dhn\" (UID: \"d957765a-bd0b-4764-9abc-23a64ba4d6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:19.404929 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.404783 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d957765a-bd0b-4764-9abc-23a64ba4d6d3-dshm\") pod \"e2e-distinct-2-simulated-kserve-859856b874-78dhn\" (UID: \"d957765a-bd0b-4764-9abc-23a64ba4d6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:19.404929 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.404842 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d957765a-bd0b-4764-9abc-23a64ba4d6d3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-859856b874-78dhn\" (UID: \"d957765a-bd0b-4764-9abc-23a64ba4d6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:19.405035 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.404921 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d957765a-bd0b-4764-9abc-23a64ba4d6d3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-859856b874-78dhn\" (UID: \"d957765a-bd0b-4764-9abc-23a64ba4d6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:19.405035 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.404953 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d957765a-bd0b-4764-9abc-23a64ba4d6d3-home\") pod \"e2e-distinct-2-simulated-kserve-859856b874-78dhn\" (UID: \"d957765a-bd0b-4764-9abc-23a64ba4d6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:19.405035 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.404971 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d957765a-bd0b-4764-9abc-23a64ba4d6d3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-859856b874-78dhn\" (UID: \"d957765a-bd0b-4764-9abc-23a64ba4d6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:19.505877 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.505851 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d957765a-bd0b-4764-9abc-23a64ba4d6d3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-859856b874-78dhn\" (UID: \"d957765a-bd0b-4764-9abc-23a64ba4d6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:19.505877 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.505882 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d957765a-bd0b-4764-9abc-23a64ba4d6d3-home\") pod \"e2e-distinct-2-simulated-kserve-859856b874-78dhn\" (UID: \"d957765a-bd0b-4764-9abc-23a64ba4d6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:19.506102 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.505975 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d957765a-bd0b-4764-9abc-23a64ba4d6d3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-859856b874-78dhn\" (UID: \"d957765a-bd0b-4764-9abc-23a64ba4d6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:19.506102 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.506032 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5prx\" (UniqueName: \"kubernetes.io/projected/d957765a-bd0b-4764-9abc-23a64ba4d6d3-kube-api-access-x5prx\") pod \"e2e-distinct-2-simulated-kserve-859856b874-78dhn\" (UID: \"d957765a-bd0b-4764-9abc-23a64ba4d6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:19.506102 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.506091 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d957765a-bd0b-4764-9abc-23a64ba4d6d3-dshm\") pod \"e2e-distinct-2-simulated-kserve-859856b874-78dhn\" (UID: \"d957765a-bd0b-4764-9abc-23a64ba4d6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:19.506246 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.506156 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d957765a-bd0b-4764-9abc-23a64ba4d6d3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-859856b874-78dhn\" (UID: \"d957765a-bd0b-4764-9abc-23a64ba4d6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:19.506297 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.506258 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d957765a-bd0b-4764-9abc-23a64ba4d6d3-home\") pod \"e2e-distinct-2-simulated-kserve-859856b874-78dhn\" (UID: \"d957765a-bd0b-4764-9abc-23a64ba4d6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:19.506346 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.506307 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d957765a-bd0b-4764-9abc-23a64ba4d6d3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-859856b874-78dhn\" (UID: \"d957765a-bd0b-4764-9abc-23a64ba4d6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:19.506478 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.506456 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d957765a-bd0b-4764-9abc-23a64ba4d6d3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-859856b874-78dhn\" (UID: \"d957765a-bd0b-4764-9abc-23a64ba4d6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:19.508265 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.508241 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d957765a-bd0b-4764-9abc-23a64ba4d6d3-dshm\") pod \"e2e-distinct-2-simulated-kserve-859856b874-78dhn\" (UID: \"d957765a-bd0b-4764-9abc-23a64ba4d6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:19.508368 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.508354 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d957765a-bd0b-4764-9abc-23a64ba4d6d3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-859856b874-78dhn\" (UID: \"d957765a-bd0b-4764-9abc-23a64ba4d6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:19.515416 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.515378 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5prx\" (UniqueName: \"kubernetes.io/projected/d957765a-bd0b-4764-9abc-23a64ba4d6d3-kube-api-access-x5prx\") pod \"e2e-distinct-2-simulated-kserve-859856b874-78dhn\" (UID: \"d957765a-bd0b-4764-9abc-23a64ba4d6d3\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:19.603493 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.603424 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:19.874116 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:19.874048 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn"] Sep 09 21:04:19.878480 ip-10-0-137-19 kubenswrapper[2580]: W0909 21:04:19.878455 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd957765a_bd0b_4764_9abc_23a64ba4d6d3.slice/crio-abde5dd048a2999fbd9ca9ff2a0ea80fe065c0d55219b1874681b9b04eff2986 WatchSource:0}: Error finding container abde5dd048a2999fbd9ca9ff2a0ea80fe065c0d55219b1874681b9b04eff2986: Status 404 returned error can't find the container with id abde5dd048a2999fbd9ca9ff2a0ea80fe065c0d55219b1874681b9b04eff2986 Sep 09 21:04:20.044586 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:20.044556 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" event={"ID":"d957765a-bd0b-4764-9abc-23a64ba4d6d3","Type":"ContainerStarted","Data":"abde5dd048a2999fbd9ca9ff2a0ea80fe065c0d55219b1874681b9b04eff2986"} Sep 09 21:04:21.049964 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:21.049939 2580 generic.go:356] "Generic (PLEG): container finished" podID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" containerID="b2c8fd21c47fae3d7ecd0dbb75d872f7d517119dec2d7ce05bf0596cd96bc97d" exitCode=0 Sep 09 21:04:21.050286 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:21.049992 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" event={"ID":"d957765a-bd0b-4764-9abc-23a64ba4d6d3","Type":"ContainerDied","Data":"b2c8fd21c47fae3d7ecd0dbb75d872f7d517119dec2d7ce05bf0596cd96bc97d"} Sep 09 21:04:21.050286 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:21.050221 2580 scope.go:117] "RemoveContainer" containerID="b2c8fd21c47fae3d7ecd0dbb75d872f7d517119dec2d7ce05bf0596cd96bc97d" Sep 09 21:04:22.056473 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:22.056444 2580 generic.go:356] "Generic (PLEG): container finished" podID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" containerID="050b2c442a1ed615aa910300ce2dc4d4bca9a1109157653431cb6a3504293d14" exitCode=0 Sep 09 21:04:22.056974 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:22.056503 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" event={"ID":"d957765a-bd0b-4764-9abc-23a64ba4d6d3","Type":"ContainerDied","Data":"050b2c442a1ed615aa910300ce2dc4d4bca9a1109157653431cb6a3504293d14"} Sep 09 21:04:22.056974 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:22.056544 2580 scope.go:117] "RemoveContainer" containerID="b2c8fd21c47fae3d7ecd0dbb75d872f7d517119dec2d7ce05bf0596cd96bc97d" Sep 09 21:04:22.056974 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:22.056767 2580 scope.go:117] "RemoveContainer" containerID="050b2c442a1ed615aa910300ce2dc4d4bca9a1109157653431cb6a3504293d14" Sep 09 21:04:22.057141 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:04:22.057004 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:04:23.015502 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:23.015469 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:04:23.061855 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:23.061833 2580 scope.go:117] "RemoveContainer" containerID="050b2c442a1ed615aa910300ce2dc4d4bca9a1109157653431cb6a3504293d14" Sep 09 21:04:23.062186 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:04:23.061994 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:04:29.604519 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:29.604479 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:29.604926 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:29.604539 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:29.604926 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:29.604555 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:29.604926 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:29.604568 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:29.604926 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:29.604870 2580 scope.go:117] "RemoveContainer" containerID="050b2c442a1ed615aa910300ce2dc4d4bca9a1109157653431cb6a3504293d14" Sep 09 21:04:29.605057 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:04:29.605039 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:04:31.669398 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:31.669367 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8"] Sep 09 21:04:31.674143 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:31.674121 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" Sep 09 21:04:31.676323 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:31.676307 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 09 21:04:31.681823 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:31.681795 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8"] Sep 09 21:04:31.769885 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:31.769841 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:04:31.795986 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:31.795960 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8\" (UID: \"cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" Sep 09 21:04:31.796138 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:31.796020 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bqdv\" (UniqueName: \"kubernetes.io/projected/cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb-kube-api-access-2bqdv\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8\" (UID: \"cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" Sep 09 21:04:31.796138 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:31.796049 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8\" (UID: \"cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" Sep 09 21:04:31.796138 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:31.796078 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8\" (UID: \"cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" Sep 09 21:04:31.796138 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:31.796126 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8\" (UID: \"cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" Sep 09 21:04:31.796306 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:31.796158 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8\" (UID: \"cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" Sep 09 21:04:31.896989 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:31.896956 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8\" (UID: \"cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" Sep 09 21:04:31.897150 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:31.896995 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8\" (UID: \"cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" Sep 09 21:04:31.897150 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:31.897016 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8\" (UID: \"cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" Sep 09 21:04:31.897150 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:31.897063 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8\" (UID: \"cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" Sep 09 21:04:31.897150 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:31.897100 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bqdv\" (UniqueName: \"kubernetes.io/projected/cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb-kube-api-access-2bqdv\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8\" (UID: \"cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" Sep 09 21:04:31.897150 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:31.897127 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8\" (UID: \"cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" Sep 09 21:04:31.897414 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:31.897353 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8\" (UID: \"cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" Sep 09 21:04:31.897414 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:31.897397 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8\" (UID: \"cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" Sep 09 21:04:31.897508 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:31.897457 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8\" (UID: \"cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" Sep 09 21:04:31.899303 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:31.899282 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8\" (UID: \"cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" Sep 09 21:04:31.899616 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:31.899598 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8\" (UID: \"cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" Sep 09 21:04:31.905906 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:31.905867 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bqdv\" (UniqueName: \"kubernetes.io/projected/cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb-kube-api-access-2bqdv\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8\" (UID: \"cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" Sep 09 21:04:31.984849 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:31.984821 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" Sep 09 21:04:32.261393 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:32.261367 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8"] Sep 09 21:04:32.263339 ip-10-0-137-19 kubenswrapper[2580]: W0909 21:04:32.263315 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2a659f_3bf3_49ae_8d8f_2a1e153d81cb.slice/crio-7a63e23ff0b4e5cdfb4620dfbc8337fe3cacba925be1a0e9078934eebb2c05cf WatchSource:0}: Error finding container 7a63e23ff0b4e5cdfb4620dfbc8337fe3cacba925be1a0e9078934eebb2c05cf: Status 404 returned error can't find the container with id 7a63e23ff0b4e5cdfb4620dfbc8337fe3cacba925be1a0e9078934eebb2c05cf Sep 09 21:04:33.105052 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:33.105019 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" event={"ID":"cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb","Type":"ContainerStarted","Data":"e4c5c02744346e5e03133235af520d4dee186688822ef6bbbdd7f9ebeab87b17"} Sep 09 21:04:33.105458 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:33.105076 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" event={"ID":"cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb","Type":"ContainerStarted","Data":"7a63e23ff0b4e5cdfb4620dfbc8337fe3cacba925be1a0e9078934eebb2c05cf"} Sep 09 21:04:33.144904 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:33.144839 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" podStartSLOduration=1.969525112 podStartE2EDuration="2.144823795s" totalImagesPullingTime="175.298683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:04:33.143320185 +0000 UTC m=+850.603962107" watchObservedRunningTime="2026-09-09 21:04:33.144823795 +0000 UTC m=+850.605465716" Sep 09 21:04:41.038806 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:41.038777 2580 scope.go:117] "RemoveContainer" containerID="050b2c442a1ed615aa910300ce2dc4d4bca9a1109157653431cb6a3504293d14" Sep 09 21:04:41.985702 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:41.985667 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" Sep 09 21:04:41.985850 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:41.985727 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" Sep 09 21:04:41.997459 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:41.997435 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" Sep 09 21:04:42.145570 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:42.145547 2580 generic.go:356] "Generic (PLEG): container finished" podID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" containerID="19faf503c1000da752ae44c7418d31e54e7220c04a85c5e9a7a4c92fc7b55314" exitCode=0 Sep 09 21:04:42.145925 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:42.145620 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" event={"ID":"d957765a-bd0b-4764-9abc-23a64ba4d6d3","Type":"ContainerDied","Data":"19faf503c1000da752ae44c7418d31e54e7220c04a85c5e9a7a4c92fc7b55314"} Sep 09 21:04:42.145925 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:42.145665 2580 scope.go:117] "RemoveContainer" containerID="050b2c442a1ed615aa910300ce2dc4d4bca9a1109157653431cb6a3504293d14" Sep 09 21:04:42.146208 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:42.146185 2580 scope.go:117] "RemoveContainer" containerID="19faf503c1000da752ae44c7418d31e54e7220c04a85c5e9a7a4c92fc7b55314" Sep 09 21:04:42.146510 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:04:42.146354 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:04:42.157139 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:42.157122 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8" Sep 09 21:04:42.402165 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:42.402078 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:04:42.818732 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:42.818694 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:04:47.065792 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:47.065759 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:04:49.603968 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:49.603929 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:49.604334 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:49.604001 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:49.604334 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:49.604028 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:49.604334 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:49.604047 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:04:49.604519 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:49.604497 2580 scope.go:117] "RemoveContainer" containerID="19faf503c1000da752ae44c7418d31e54e7220c04a85c5e9a7a4c92fc7b55314" Sep 09 21:04:49.604772 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:04:49.604747 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:04:58.781261 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:04:58.781216 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:05:03.040102 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:03.040067 2580 scope.go:117] "RemoveContainer" containerID="19faf503c1000da752ae44c7418d31e54e7220c04a85c5e9a7a4c92fc7b55314" Sep 09 21:05:04.243811 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:04.243780 2580 generic.go:356] "Generic (PLEG): container finished" podID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" containerID="59bc311961843c4b5a5f877782fb26d11f3ae209a64d67128dd04686e987b94e" exitCode=0 Sep 09 21:05:04.244221 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:04.243837 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" event={"ID":"d957765a-bd0b-4764-9abc-23a64ba4d6d3","Type":"ContainerDied","Data":"59bc311961843c4b5a5f877782fb26d11f3ae209a64d67128dd04686e987b94e"} Sep 09 21:05:04.244539 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:04.243912 2580 scope.go:117] "RemoveContainer" containerID="19faf503c1000da752ae44c7418d31e54e7220c04a85c5e9a7a4c92fc7b55314" Sep 09 21:05:04.244626 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:04.244613 2580 scope.go:117] "RemoveContainer" containerID="59bc311961843c4b5a5f877782fb26d11f3ae209a64d67128dd04686e987b94e" Sep 09 21:05:04.244831 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:05:04.244809 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:05:05.170365 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:05.170332 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp"] Sep 09 21:05:05.174909 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:05.174873 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:05.177114 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:05.177095 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 09 21:05:05.190026 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:05.190006 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp"] Sep 09 21:05:05.259705 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:05.259680 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:05:05.361254 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:05.361229 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d7e7307-f923-4b20-9596-9da76d196e94-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-8chmp\" (UID: \"4d7e7307-f923-4b20-9596-9da76d196e94\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:05.361406 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:05.361267 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d7e7307-f923-4b20-9596-9da76d196e94-home\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-8chmp\" (UID: \"4d7e7307-f923-4b20-9596-9da76d196e94\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:05.361406 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:05.361294 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rscmx\" (UniqueName: \"kubernetes.io/projected/4d7e7307-f923-4b20-9596-9da76d196e94-kube-api-access-rscmx\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-8chmp\" (UID: \"4d7e7307-f923-4b20-9596-9da76d196e94\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:05.361406 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:05.361371 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d7e7307-f923-4b20-9596-9da76d196e94-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-8chmp\" (UID: \"4d7e7307-f923-4b20-9596-9da76d196e94\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:05.361406 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:05.361405 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d7e7307-f923-4b20-9596-9da76d196e94-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-8chmp\" (UID: \"4d7e7307-f923-4b20-9596-9da76d196e94\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:05.361581 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:05.361440 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d7e7307-f923-4b20-9596-9da76d196e94-dshm\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-8chmp\" (UID: \"4d7e7307-f923-4b20-9596-9da76d196e94\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:05.462361 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:05.462336 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d7e7307-f923-4b20-9596-9da76d196e94-home\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-8chmp\" (UID: \"4d7e7307-f923-4b20-9596-9da76d196e94\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:05.462504 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:05.462369 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rscmx\" (UniqueName: \"kubernetes.io/projected/4d7e7307-f923-4b20-9596-9da76d196e94-kube-api-access-rscmx\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-8chmp\" (UID: \"4d7e7307-f923-4b20-9596-9da76d196e94\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:05.462504 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:05.462393 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d7e7307-f923-4b20-9596-9da76d196e94-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-8chmp\" (UID: \"4d7e7307-f923-4b20-9596-9da76d196e94\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:05.462504 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:05.462414 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d7e7307-f923-4b20-9596-9da76d196e94-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-8chmp\" (UID: \"4d7e7307-f923-4b20-9596-9da76d196e94\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:05.462504 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:05.462440 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d7e7307-f923-4b20-9596-9da76d196e94-dshm\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-8chmp\" (UID: \"4d7e7307-f923-4b20-9596-9da76d196e94\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:05.462504 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:05.462474 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d7e7307-f923-4b20-9596-9da76d196e94-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-8chmp\" (UID: \"4d7e7307-f923-4b20-9596-9da76d196e94\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:05.462773 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:05.462751 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d7e7307-f923-4b20-9596-9da76d196e94-home\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-8chmp\" (UID: \"4d7e7307-f923-4b20-9596-9da76d196e94\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:05.462773 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:05.462767 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d7e7307-f923-4b20-9596-9da76d196e94-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-8chmp\" (UID: \"4d7e7307-f923-4b20-9596-9da76d196e94\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:05.462876 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:05.462817 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d7e7307-f923-4b20-9596-9da76d196e94-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-8chmp\" (UID: \"4d7e7307-f923-4b20-9596-9da76d196e94\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:05.464780 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:05.464761 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d7e7307-f923-4b20-9596-9da76d196e94-dshm\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-8chmp\" (UID: \"4d7e7307-f923-4b20-9596-9da76d196e94\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:05.464957 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:05.464941 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d7e7307-f923-4b20-9596-9da76d196e94-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-8chmp\" (UID: \"4d7e7307-f923-4b20-9596-9da76d196e94\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:05.470927 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:05.470910 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rscmx\" (UniqueName: \"kubernetes.io/projected/4d7e7307-f923-4b20-9596-9da76d196e94-kube-api-access-rscmx\") pod \"premium-simulated-simulated-premium-kserve-5758cf6949-8chmp\" (UID: \"4d7e7307-f923-4b20-9596-9da76d196e94\") " pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:05.483781 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:05.483759 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:05.764790 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:05.764768 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp"] Sep 09 21:05:05.767009 ip-10-0-137-19 kubenswrapper[2580]: W0909 21:05:05.766985 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d7e7307_f923_4b20_9596_9da76d196e94.slice/crio-668f8f122dc0688251f7815e33a6b06850b72bb3f139a0e36bdfb039df80dd78 WatchSource:0}: Error finding container 668f8f122dc0688251f7815e33a6b06850b72bb3f139a0e36bdfb039df80dd78: Status 404 returned error can't find the container with id 668f8f122dc0688251f7815e33a6b06850b72bb3f139a0e36bdfb039df80dd78 Sep 09 21:05:06.256268 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:06.256235 2580 generic.go:356] "Generic (PLEG): container finished" podID="4d7e7307-f923-4b20-9596-9da76d196e94" containerID="446d5ac7fb08e1de9dc69cc4bfced48b715652f3e2e94d20f6d8b6b5aefb61a8" exitCode=0 Sep 09 21:05:06.256428 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:06.256284 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" event={"ID":"4d7e7307-f923-4b20-9596-9da76d196e94","Type":"ContainerDied","Data":"446d5ac7fb08e1de9dc69cc4bfced48b715652f3e2e94d20f6d8b6b5aefb61a8"} Sep 09 21:05:06.256428 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:06.256311 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" event={"ID":"4d7e7307-f923-4b20-9596-9da76d196e94","Type":"ContainerStarted","Data":"668f8f122dc0688251f7815e33a6b06850b72bb3f139a0e36bdfb039df80dd78"} Sep 09 21:05:06.256605 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:06.256588 2580 scope.go:117] "RemoveContainer" containerID="446d5ac7fb08e1de9dc69cc4bfced48b715652f3e2e94d20f6d8b6b5aefb61a8" Sep 09 21:05:07.265574 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:07.265545 2580 generic.go:356] "Generic (PLEG): container finished" podID="4d7e7307-f923-4b20-9596-9da76d196e94" containerID="57e26acde58082f90f7e5e67f207b7dc50cecb3032ffb7f0f6077d2ee6806197" exitCode=0 Sep 09 21:05:07.266002 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:07.265635 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" event={"ID":"4d7e7307-f923-4b20-9596-9da76d196e94","Type":"ContainerDied","Data":"57e26acde58082f90f7e5e67f207b7dc50cecb3032ffb7f0f6077d2ee6806197"} Sep 09 21:05:07.266002 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:07.265681 2580 scope.go:117] "RemoveContainer" containerID="446d5ac7fb08e1de9dc69cc4bfced48b715652f3e2e94d20f6d8b6b5aefb61a8" Sep 09 21:05:07.266002 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:07.265864 2580 scope.go:117] "RemoveContainer" containerID="57e26acde58082f90f7e5e67f207b7dc50cecb3032ffb7f0f6077d2ee6806197" Sep 09 21:05:07.266125 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:05:07.266104 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_llm(4d7e7307-f923-4b20-9596-9da76d196e94)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" podUID="4d7e7307-f923-4b20-9596-9da76d196e94" Sep 09 21:05:08.271358 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:08.271334 2580 scope.go:117] "RemoveContainer" containerID="57e26acde58082f90f7e5e67f207b7dc50cecb3032ffb7f0f6077d2ee6806197" Sep 09 21:05:08.271722 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:05:08.271487 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_llm(4d7e7307-f923-4b20-9596-9da76d196e94)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" podUID="4d7e7307-f923-4b20-9596-9da76d196e94" Sep 09 21:05:09.603995 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:09.603958 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:05:09.604393 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:09.604019 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:05:09.604393 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:09.604035 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:05:09.604393 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:09.604048 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:05:09.604393 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:09.604356 2580 scope.go:117] "RemoveContainer" containerID="59bc311961843c4b5a5f877782fb26d11f3ae209a64d67128dd04686e987b94e" Sep 09 21:05:09.604589 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:05:09.604518 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:05:15.484819 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:15.484735 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:15.484819 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:15.484792 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:15.484819 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:15.484805 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:15.484819 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:15.484814 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:15.485294 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:15.485276 2580 scope.go:117] "RemoveContainer" containerID="57e26acde58082f90f7e5e67f207b7dc50cecb3032ffb7f0f6077d2ee6806197" Sep 09 21:05:15.485510 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:05:15.485487 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_llm(4d7e7307-f923-4b20-9596-9da76d196e94)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" podUID="4d7e7307-f923-4b20-9596-9da76d196e94" Sep 09 21:05:21.037742 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:21.037715 2580 scope.go:117] "RemoveContainer" containerID="59bc311961843c4b5a5f877782fb26d11f3ae209a64d67128dd04686e987b94e" Sep 09 21:05:21.038116 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:05:21.037862 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:05:22.994298 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:22.994266 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tctqv_ddb3d74a-a1bb-4bda-8d0d-89320e50f357/ovn-acl-logging/0.log" Sep 09 21:05:22.995843 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:22.995822 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tctqv_ddb3d74a-a1bb-4bda-8d0d-89320e50f357/ovn-acl-logging/0.log" Sep 09 21:05:30.038580 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:30.038550 2580 scope.go:117] "RemoveContainer" containerID="57e26acde58082f90f7e5e67f207b7dc50cecb3032ffb7f0f6077d2ee6806197" Sep 09 21:05:30.367378 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:30.367348 2580 generic.go:356] "Generic (PLEG): container finished" podID="4d7e7307-f923-4b20-9596-9da76d196e94" containerID="2adff943ee59faefe099ca68e377bc9c39dc17b33aa76bf2409d39fca0ee96e4" exitCode=0 Sep 09 21:05:30.367487 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:30.367416 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" event={"ID":"4d7e7307-f923-4b20-9596-9da76d196e94","Type":"ContainerDied","Data":"2adff943ee59faefe099ca68e377bc9c39dc17b33aa76bf2409d39fca0ee96e4"} Sep 09 21:05:30.367487 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:30.367461 2580 scope.go:117] "RemoveContainer" containerID="57e26acde58082f90f7e5e67f207b7dc50cecb3032ffb7f0f6077d2ee6806197" Sep 09 21:05:30.367800 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:30.367783 2580 scope.go:117] "RemoveContainer" containerID="2adff943ee59faefe099ca68e377bc9c39dc17b33aa76bf2409d39fca0ee96e4" Sep 09 21:05:30.367985 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:05:30.367964 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_llm(4d7e7307-f923-4b20-9596-9da76d196e94)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" podUID="4d7e7307-f923-4b20-9596-9da76d196e94" Sep 09 21:05:34.038461 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:34.038424 2580 scope.go:117] "RemoveContainer" containerID="59bc311961843c4b5a5f877782fb26d11f3ae209a64d67128dd04686e987b94e" Sep 09 21:05:34.038966 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:05:34.038620 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:05:35.484879 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:35.484847 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:35.485260 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:35.484907 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:35.485260 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:35.484918 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:35.485260 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:35.484925 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:05:35.485260 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:35.485183 2580 scope.go:117] "RemoveContainer" containerID="2adff943ee59faefe099ca68e377bc9c39dc17b33aa76bf2409d39fca0ee96e4" Sep 09 21:05:35.485394 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:05:35.485319 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_llm(4d7e7307-f923-4b20-9596-9da76d196e94)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" podUID="4d7e7307-f923-4b20-9596-9da76d196e94" Sep 09 21:05:45.038358 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:45.038324 2580 scope.go:117] "RemoveContainer" containerID="59bc311961843c4b5a5f877782fb26d11f3ae209a64d67128dd04686e987b94e" Sep 09 21:05:45.038804 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:45.038671 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 21:05:45.446220 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:45.446195 2580 generic.go:356] "Generic (PLEG): container finished" podID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" containerID="f5f39527e24dc9c80ce32833e8dc15abe8e7fdd3c99bb8ec1fa3aca5ee9a56ff" exitCode=0 Sep 09 21:05:45.446324 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:45.446255 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" event={"ID":"d957765a-bd0b-4764-9abc-23a64ba4d6d3","Type":"ContainerDied","Data":"f5f39527e24dc9c80ce32833e8dc15abe8e7fdd3c99bb8ec1fa3aca5ee9a56ff"} Sep 09 21:05:45.446324 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:45.446303 2580 scope.go:117] "RemoveContainer" containerID="59bc311961843c4b5a5f877782fb26d11f3ae209a64d67128dd04686e987b94e" Sep 09 21:05:45.446649 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:45.446627 2580 scope.go:117] "RemoveContainer" containerID="f5f39527e24dc9c80ce32833e8dc15abe8e7fdd3c99bb8ec1fa3aca5ee9a56ff" Sep 09 21:05:45.446874 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:05:45.446848 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:05:48.038193 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:48.038165 2580 scope.go:117] "RemoveContainer" containerID="2adff943ee59faefe099ca68e377bc9c39dc17b33aa76bf2409d39fca0ee96e4" Sep 09 21:05:48.038544 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:05:48.038356 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_llm(4d7e7307-f923-4b20-9596-9da76d196e94)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" podUID="4d7e7307-f923-4b20-9596-9da76d196e94" Sep 09 21:05:49.603941 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:49.603911 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:05:49.604300 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:49.603972 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:05:49.604300 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:49.603991 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:05:49.604300 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:49.604003 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:05:49.604406 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:05:49.604319 2580 scope.go:117] "RemoveContainer" containerID="f5f39527e24dc9c80ce32833e8dc15abe8e7fdd3c99bb8ec1fa3aca5ee9a56ff" Sep 09 21:05:49.604507 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:05:49.604490 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:06:02.038457 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:06:02.038426 2580 scope.go:117] "RemoveContainer" containerID="f5f39527e24dc9c80ce32833e8dc15abe8e7fdd3c99bb8ec1fa3aca5ee9a56ff" Sep 09 21:06:02.039018 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:06:02.038639 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:06:03.040044 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:06:03.040020 2580 scope.go:117] "RemoveContainer" containerID="2adff943ee59faefe099ca68e377bc9c39dc17b33aa76bf2409d39fca0ee96e4" Sep 09 21:06:03.527611 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:06:03.527585 2580 generic.go:356] "Generic (PLEG): container finished" podID="4d7e7307-f923-4b20-9596-9da76d196e94" containerID="af2f2333000d4965793aa14946e6383fbf3768365408ea1074d70992a2f1531b" exitCode=0 Sep 09 21:06:03.527728 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:06:03.527630 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" event={"ID":"4d7e7307-f923-4b20-9596-9da76d196e94","Type":"ContainerDied","Data":"af2f2333000d4965793aa14946e6383fbf3768365408ea1074d70992a2f1531b"} Sep 09 21:06:03.527728 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:06:03.527668 2580 scope.go:117] "RemoveContainer" containerID="2adff943ee59faefe099ca68e377bc9c39dc17b33aa76bf2409d39fca0ee96e4" Sep 09 21:06:03.527993 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:06:03.527970 2580 scope.go:117] "RemoveContainer" containerID="af2f2333000d4965793aa14946e6383fbf3768365408ea1074d70992a2f1531b" Sep 09 21:06:03.528180 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:06:03.528155 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_llm(4d7e7307-f923-4b20-9596-9da76d196e94)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" podUID="4d7e7307-f923-4b20-9596-9da76d196e94" Sep 09 21:06:05.484454 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:06:05.484421 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:06:05.484814 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:06:05.484463 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:06:05.484814 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:06:05.484474 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:06:05.484814 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:06:05.484483 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:06:05.484814 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:06:05.484732 2580 scope.go:117] "RemoveContainer" containerID="af2f2333000d4965793aa14946e6383fbf3768365408ea1074d70992a2f1531b" Sep 09 21:06:05.484984 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:06:05.484866 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_llm(4d7e7307-f923-4b20-9596-9da76d196e94)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" podUID="4d7e7307-f923-4b20-9596-9da76d196e94" Sep 09 21:06:13.040519 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:06:13.040484 2580 scope.go:117] "RemoveContainer" containerID="f5f39527e24dc9c80ce32833e8dc15abe8e7fdd3c99bb8ec1fa3aca5ee9a56ff" Sep 09 21:06:13.040978 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:06:13.040706 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:06:17.705977 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:06:17.705945 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:06:18.037674 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:06:18.037608 2580 scope.go:117] "RemoveContainer" containerID="af2f2333000d4965793aa14946e6383fbf3768365408ea1074d70992a2f1531b" Sep 09 21:06:18.037796 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:06:18.037770 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_llm(4d7e7307-f923-4b20-9596-9da76d196e94)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" podUID="4d7e7307-f923-4b20-9596-9da76d196e94" Sep 09 21:06:27.038157 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:06:27.038125 2580 scope.go:117] "RemoveContainer" containerID="f5f39527e24dc9c80ce32833e8dc15abe8e7fdd3c99bb8ec1fa3aca5ee9a56ff" Sep 09 21:06:27.038506 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:06:27.038290 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:06:32.038611 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:06:32.038585 2580 scope.go:117] "RemoveContainer" containerID="af2f2333000d4965793aa14946e6383fbf3768365408ea1074d70992a2f1531b" Sep 09 21:06:32.038997 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:06:32.038731 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_llm(4d7e7307-f923-4b20-9596-9da76d196e94)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" podUID="4d7e7307-f923-4b20-9596-9da76d196e94" Sep 09 21:06:40.038533 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:06:40.038506 2580 scope.go:117] "RemoveContainer" containerID="f5f39527e24dc9c80ce32833e8dc15abe8e7fdd3c99bb8ec1fa3aca5ee9a56ff" Sep 09 21:06:40.039049 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:06:40.038658 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:06:43.040906 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:06:43.040874 2580 scope.go:117] "RemoveContainer" containerID="af2f2333000d4965793aa14946e6383fbf3768365408ea1074d70992a2f1531b" Sep 09 21:06:43.041254 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:06:43.041031 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_llm(4d7e7307-f923-4b20-9596-9da76d196e94)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" podUID="4d7e7307-f923-4b20-9596-9da76d196e94" Sep 09 21:06:52.038483 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:06:52.038406 2580 scope.go:117] "RemoveContainer" containerID="f5f39527e24dc9c80ce32833e8dc15abe8e7fdd3c99bb8ec1fa3aca5ee9a56ff" Sep 09 21:06:52.038838 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:06:52.038562 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:06:57.038468 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:06:57.038439 2580 scope.go:117] "RemoveContainer" containerID="af2f2333000d4965793aa14946e6383fbf3768365408ea1074d70992a2f1531b" Sep 09 21:06:57.762476 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:06:57.762449 2580 generic.go:356] "Generic (PLEG): container finished" podID="4d7e7307-f923-4b20-9596-9da76d196e94" containerID="78985ebbf243e38c03023dca965b340e6245ac370a663840fcd064ec2dab3bf3" exitCode=0 Sep 09 21:06:57.762603 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:06:57.762523 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" event={"ID":"4d7e7307-f923-4b20-9596-9da76d196e94","Type":"ContainerDied","Data":"78985ebbf243e38c03023dca965b340e6245ac370a663840fcd064ec2dab3bf3"} Sep 09 21:06:57.762603 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:06:57.762570 2580 scope.go:117] "RemoveContainer" containerID="af2f2333000d4965793aa14946e6383fbf3768365408ea1074d70992a2f1531b" Sep 09 21:06:57.762904 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:06:57.762874 2580 scope.go:117] "RemoveContainer" containerID="78985ebbf243e38c03023dca965b340e6245ac370a663840fcd064ec2dab3bf3" Sep 09 21:06:57.763066 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:06:57.763050 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_llm(4d7e7307-f923-4b20-9596-9da76d196e94)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" podUID="4d7e7307-f923-4b20-9596-9da76d196e94" Sep 09 21:07:03.039790 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:07:03.039761 2580 scope.go:117] "RemoveContainer" containerID="f5f39527e24dc9c80ce32833e8dc15abe8e7fdd3c99bb8ec1fa3aca5ee9a56ff" Sep 09 21:07:03.040243 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:07:03.039970 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:07:03.946460 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:07:03.946426 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:07:05.484837 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:07:05.484809 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:07:05.485200 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:07:05.484857 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:07:05.485200 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:07:05.484869 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:07:05.485200 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:07:05.484878 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:07:05.485303 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:07:05.485192 2580 scope.go:117] "RemoveContainer" containerID="78985ebbf243e38c03023dca965b340e6245ac370a663840fcd064ec2dab3bf3" Sep 09 21:07:05.485359 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:07:05.485340 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_llm(4d7e7307-f923-4b20-9596-9da76d196e94)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" podUID="4d7e7307-f923-4b20-9596-9da76d196e94" Sep 09 21:07:16.038590 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:07:16.038555 2580 scope.go:117] "RemoveContainer" containerID="f5f39527e24dc9c80ce32833e8dc15abe8e7fdd3c99bb8ec1fa3aca5ee9a56ff" Sep 09 21:07:16.847109 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:07:16.847079 2580 generic.go:356] "Generic (PLEG): container finished" podID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" containerID="e55767363f11486a984eee5494d7a842662273043d124fd83aa81a39a84f82fa" exitCode=0 Sep 09 21:07:16.847243 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:07:16.847146 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" event={"ID":"d957765a-bd0b-4764-9abc-23a64ba4d6d3","Type":"ContainerDied","Data":"e55767363f11486a984eee5494d7a842662273043d124fd83aa81a39a84f82fa"} Sep 09 21:07:16.847243 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:07:16.847206 2580 scope.go:117] "RemoveContainer" containerID="f5f39527e24dc9c80ce32833e8dc15abe8e7fdd3c99bb8ec1fa3aca5ee9a56ff" Sep 09 21:07:16.847480 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:07:16.847465 2580 scope.go:117] "RemoveContainer" containerID="e55767363f11486a984eee5494d7a842662273043d124fd83aa81a39a84f82fa" Sep 09 21:07:16.847646 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:07:16.847630 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:07:19.603647 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:07:19.603609 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:07:19.604137 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:07:19.603674 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:07:19.604137 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:07:19.603692 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:07:19.604137 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:07:19.603706 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" Sep 09 21:07:19.604137 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:07:19.604111 2580 scope.go:117] "RemoveContainer" containerID="e55767363f11486a984eee5494d7a842662273043d124fd83aa81a39a84f82fa" Sep 09 21:07:19.604351 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:07:19.604327 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:07:19.862107 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:07:19.862033 2580 scope.go:117] "RemoveContainer" containerID="e55767363f11486a984eee5494d7a842662273043d124fd83aa81a39a84f82fa" Sep 09 21:07:19.862243 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:07:19.862222 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:07:21.038513 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:07:21.038478 2580 scope.go:117] "RemoveContainer" containerID="78985ebbf243e38c03023dca965b340e6245ac370a663840fcd064ec2dab3bf3" Sep 09 21:07:21.038882 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:07:21.038661 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_llm(4d7e7307-f923-4b20-9596-9da76d196e94)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" podUID="4d7e7307-f923-4b20-9596-9da76d196e94" Sep 09 21:07:35.038279 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:07:35.038247 2580 scope.go:117] "RemoveContainer" containerID="78985ebbf243e38c03023dca965b340e6245ac370a663840fcd064ec2dab3bf3" Sep 09 21:07:35.038736 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:07:35.038380 2580 scope.go:117] "RemoveContainer" containerID="e55767363f11486a984eee5494d7a842662273043d124fd83aa81a39a84f82fa" Sep 09 21:07:35.038736 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:07:35.038393 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_llm(4d7e7307-f923-4b20-9596-9da76d196e94)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" podUID="4d7e7307-f923-4b20-9596-9da76d196e94" Sep 09 21:07:35.038736 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:07:35.038557 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:07:48.038162 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:07:48.038133 2580 scope.go:117] "RemoveContainer" containerID="78985ebbf243e38c03023dca965b340e6245ac370a663840fcd064ec2dab3bf3" Sep 09 21:07:48.038617 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:07:48.038280 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_llm(4d7e7307-f923-4b20-9596-9da76d196e94)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" podUID="4d7e7307-f923-4b20-9596-9da76d196e94" Sep 09 21:07:49.037948 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:07:49.037922 2580 scope.go:117] "RemoveContainer" containerID="e55767363f11486a984eee5494d7a842662273043d124fd83aa81a39a84f82fa" Sep 09 21:07:49.038100 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:07:49.038073 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:08:00.409751 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:08:00.409712 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:08:01.038139 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:08:01.038109 2580 scope.go:117] "RemoveContainer" containerID="78985ebbf243e38c03023dca965b340e6245ac370a663840fcd064ec2dab3bf3" Sep 09 21:08:01.038290 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:08:01.038259 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_llm(4d7e7307-f923-4b20-9596-9da76d196e94)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" podUID="4d7e7307-f923-4b20-9596-9da76d196e94" Sep 09 21:08:03.039885 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:08:03.039845 2580 scope.go:117] "RemoveContainer" containerID="e55767363f11486a984eee5494d7a842662273043d124fd83aa81a39a84f82fa" Sep 09 21:08:03.040343 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:08:03.040048 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:08:14.038318 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:08:14.038283 2580 scope.go:117] "RemoveContainer" containerID="78985ebbf243e38c03023dca965b340e6245ac370a663840fcd064ec2dab3bf3" Sep 09 21:08:14.038760 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:08:14.038425 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_llm(4d7e7307-f923-4b20-9596-9da76d196e94)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" podUID="4d7e7307-f923-4b20-9596-9da76d196e94" Sep 09 21:08:18.037869 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:08:18.037840 2580 scope.go:117] "RemoveContainer" containerID="e55767363f11486a984eee5494d7a842662273043d124fd83aa81a39a84f82fa" Sep 09 21:08:18.038231 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:08:18.038026 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:08:28.038113 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:08:28.038084 2580 scope.go:117] "RemoveContainer" containerID="78985ebbf243e38c03023dca965b340e6245ac370a663840fcd064ec2dab3bf3" Sep 09 21:08:29.038038 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:08:29.038015 2580 scope.go:117] "RemoveContainer" containerID="e55767363f11486a984eee5494d7a842662273043d124fd83aa81a39a84f82fa" Sep 09 21:08:29.038181 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:08:29.038164 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:08:29.146499 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:08:29.146473 2580 generic.go:356] "Generic (PLEG): container finished" podID="4d7e7307-f923-4b20-9596-9da76d196e94" containerID="c875dd5c204bdb8440911137eb2a5a41331ecff8dae88e83d1e14ef3314f4729" exitCode=0 Sep 09 21:08:29.146639 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:08:29.146549 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" event={"ID":"4d7e7307-f923-4b20-9596-9da76d196e94","Type":"ContainerDied","Data":"c875dd5c204bdb8440911137eb2a5a41331ecff8dae88e83d1e14ef3314f4729"} Sep 09 21:08:29.146639 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:08:29.146603 2580 scope.go:117] "RemoveContainer" containerID="78985ebbf243e38c03023dca965b340e6245ac370a663840fcd064ec2dab3bf3" Sep 09 21:08:29.146913 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:08:29.146875 2580 scope.go:117] "RemoveContainer" containerID="c875dd5c204bdb8440911137eb2a5a41331ecff8dae88e83d1e14ef3314f4729" Sep 09 21:08:29.147066 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:08:29.147047 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_llm(4d7e7307-f923-4b20-9596-9da76d196e94)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" podUID="4d7e7307-f923-4b20-9596-9da76d196e94" Sep 09 21:08:35.484199 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:08:35.484166 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:08:35.484550 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:08:35.484219 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:08:35.484550 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:08:35.484234 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:08:35.484550 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:08:35.484247 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" Sep 09 21:08:35.484656 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:08:35.484586 2580 scope.go:117] "RemoveContainer" containerID="c875dd5c204bdb8440911137eb2a5a41331ecff8dae88e83d1e14ef3314f4729" Sep 09 21:08:35.484747 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:08:35.484729 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_llm(4d7e7307-f923-4b20-9596-9da76d196e94)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" podUID="4d7e7307-f923-4b20-9596-9da76d196e94" Sep 09 21:08:43.041301 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:08:43.041273 2580 scope.go:117] "RemoveContainer" containerID="e55767363f11486a984eee5494d7a842662273043d124fd83aa81a39a84f82fa" Sep 09 21:08:43.041671 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:08:43.041468 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:08:51.038317 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:08:51.038284 2580 scope.go:117] "RemoveContainer" containerID="c875dd5c204bdb8440911137eb2a5a41331ecff8dae88e83d1e14ef3314f4729" Sep 09 21:08:51.038762 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:08:51.038432 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_llm(4d7e7307-f923-4b20-9596-9da76d196e94)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" podUID="4d7e7307-f923-4b20-9596-9da76d196e94" Sep 09 21:08:57.038197 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:08:57.038162 2580 scope.go:117] "RemoveContainer" containerID="e55767363f11486a984eee5494d7a842662273043d124fd83aa81a39a84f82fa" Sep 09 21:08:57.038678 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:08:57.038380 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:09:05.038176 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:05.038146 2580 scope.go:117] "RemoveContainer" containerID="c875dd5c204bdb8440911137eb2a5a41331ecff8dae88e83d1e14ef3314f4729" Sep 09 21:09:05.038633 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:09:05.038346 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_llm(4d7e7307-f923-4b20-9596-9da76d196e94)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" podUID="4d7e7307-f923-4b20-9596-9da76d196e94" Sep 09 21:09:12.038277 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:12.038247 2580 scope.go:117] "RemoveContainer" containerID="e55767363f11486a984eee5494d7a842662273043d124fd83aa81a39a84f82fa" Sep 09 21:09:12.040294 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:09:12.038415 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:09:16.678033 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:16.678001 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-m6bbc_64b5633f-b9c9-4d46-9224-fd46d2e5cb01/manager/0.log" Sep 09 21:09:16.794345 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:16.794320 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-njk78_4e582a30-e52b-4a4e-9b26-f4e0f7ec57d4/manager/0.log" Sep 09 21:09:16.916009 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:16.915982 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d8ddb97f8-hzjcq_14ba4026-2fb2-4334-964f-06cc61450890/manager/0.log" Sep 09 21:09:17.531595 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:17.531561 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cf9547dff-vpfp4_a43b1176-2185-44bd-acf9-5b24408e9095/manager/0.log" Sep 09 21:09:17.540637 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:17.540616 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cf9547dff-vpfp4_a43b1176-2185-44bd-acf9-5b24408e9095/copy-manifests/0.log" Sep 09 21:09:17.858833 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:17.858760 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5c576cc74c-8fjgr_87934a77-22ad-4012-92ef-c1bada029bd0/maas-api/0.log" Sep 09 21:09:17.964895 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:17.964872 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-sb2sp_266d0b42-bcd5-4101-9876-57c67cb86bb5/postgres/0.log" Sep 09 21:09:18.836508 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:18.836484 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-p5fdj_96951f8a-31a7-4b58-ae9d-7a72a1333071/manager/0.log" Sep 09 21:09:18.945330 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:18.945305 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-h9s2p_1936b418-b1a9-4731-9fe1-b63206b10982/manager/0.log" Sep 09 21:09:20.038403 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:20.038377 2580 scope.go:117] "RemoveContainer" containerID="c875dd5c204bdb8440911137eb2a5a41331ecff8dae88e83d1e14ef3314f4729" Sep 09 21:09:20.038763 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:09:20.038533 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_llm(4d7e7307-f923-4b20-9596-9da76d196e94)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" podUID="4d7e7307-f923-4b20-9596-9da76d196e94" Sep 09 21:09:20.156354 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:20.156333 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rmpg9_4597f764-9265-44e0-b17b-660f72201d59/istio-proxy/1.log" Sep 09 21:09:20.840170 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:20.840146 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-859856b874-78dhn_d957765a-bd0b-4764-9abc-23a64ba4d6d3/main/5.log" Sep 09 21:09:21.059969 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:21.059943 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-kg5q8_cc2a659f-3bf3-49ae-8d8f-2a1e153d81cb/main/0.log" Sep 09 21:09:21.499661 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:21.499639 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_4d7e7307-f923-4b20-9596-9da76d196e94/main/5.log" Sep 09 21:09:25.038637 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:25.038607 2580 scope.go:117] "RemoveContainer" containerID="e55767363f11486a984eee5494d7a842662273043d124fd83aa81a39a84f82fa" Sep 09 21:09:25.039020 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:09:25.038774 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:09:26.457093 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:26.457058 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:09:26.550004 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:26.549974 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" limit=1024 Sep 09 21:09:29.385412 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:29.385382 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-csblg_91f58455-46ed-4617-bfe6-ef79a8fc05f3/global-pull-secret-syncer/0.log" Sep 09 21:09:29.512033 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:29.511998 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2g5nr_f5167c4c-e0f9-496f-b62f-78764ba3cbd7/konnectivity-agent/0.log" Sep 09 21:09:29.615527 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:29.615496 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-19.ec2.internal_70b70169d86c3bcf2d0a1e114ed46d9a/haproxy/0.log" Sep 09 21:09:31.037697 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:31.037664 2580 scope.go:117] "RemoveContainer" containerID="c875dd5c204bdb8440911137eb2a5a41331ecff8dae88e83d1e14ef3314f4729" Sep 09 21:09:31.038171 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:09:31.037854 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_llm(4d7e7307-f923-4b20-9596-9da76d196e94)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" podUID="4d7e7307-f923-4b20-9596-9da76d196e94" Sep 09 21:09:33.284960 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:33.284936 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-p5fdj_96951f8a-31a7-4b58-ae9d-7a72a1333071/manager/0.log" Sep 09 21:09:33.322077 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:33.322048 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-h9s2p_1936b418-b1a9-4731-9fe1-b63206b10982/manager/0.log" Sep 09 21:09:35.895607 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:35.895529 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hzjk9_57beebb6-a10e-45af-92c9-2d1a066a381e/node-exporter/0.log" Sep 09 21:09:35.919521 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:35.919498 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hzjk9_57beebb6-a10e-45af-92c9-2d1a066a381e/kube-rbac-proxy/0.log" Sep 09 21:09:35.944462 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:35.944446 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hzjk9_57beebb6-a10e-45af-92c9-2d1a066a381e/init-textfile/0.log" Sep 09 21:09:36.037770 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:36.037753 2580 scope.go:117] "RemoveContainer" containerID="e55767363f11486a984eee5494d7a842662273043d124fd83aa81a39a84f82fa" Sep 09 21:09:36.037954 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:09:36.037924 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:09:36.101537 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:36.101521 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070/prometheus/0.log" Sep 09 21:09:36.133790 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:36.133772 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070/config-reloader/0.log" Sep 09 21:09:36.160342 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:36.160298 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070/thanos-sidecar/0.log" Sep 09 21:09:36.183993 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:36.183969 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070/kube-rbac-proxy-web/0.log" Sep 09 21:09:36.208468 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:36.208449 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070/kube-rbac-proxy/0.log" Sep 09 21:09:36.231668 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:36.231635 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070/kube-rbac-proxy-thanos/0.log" Sep 09 21:09:36.255210 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:36.255178 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9c54bb0f-91a3-4aa1-bdd4-86cf6d5bb070/init-config-reloader/0.log" Sep 09 21:09:39.478919 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:39.476663 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5djh2/perf-node-gather-daemonset-x8pmg"] Sep 09 21:09:39.481546 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:39.481512 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5djh2/perf-node-gather-daemonset-x8pmg" Sep 09 21:09:39.484238 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:39.484210 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5djh2\"/\"kube-root-ca.crt\"" Sep 09 21:09:39.484365 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:39.484271 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5djh2\"/\"openshift-service-ca.crt\"" Sep 09 21:09:39.484940 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:39.484926 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5djh2\"/\"default-dockercfg-vknfd\"" Sep 09 21:09:39.494374 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:39.494354 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5djh2/perf-node-gather-daemonset-x8pmg"] Sep 09 21:09:39.659713 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:39.659686 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e3d5731-bf9e-4f2c-939b-0706ca41780b-sys\") pod \"perf-node-gather-daemonset-x8pmg\" (UID: \"0e3d5731-bf9e-4f2c-939b-0706ca41780b\") " pod="openshift-must-gather-5djh2/perf-node-gather-daemonset-x8pmg" Sep 09 21:09:39.659873 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:39.659738 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpq52\" (UniqueName: \"kubernetes.io/projected/0e3d5731-bf9e-4f2c-939b-0706ca41780b-kube-api-access-gpq52\") pod \"perf-node-gather-daemonset-x8pmg\" (UID: \"0e3d5731-bf9e-4f2c-939b-0706ca41780b\") " pod="openshift-must-gather-5djh2/perf-node-gather-daemonset-x8pmg" Sep 09 21:09:39.659873 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:39.659831 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0e3d5731-bf9e-4f2c-939b-0706ca41780b-proc\") pod \"perf-node-gather-daemonset-x8pmg\" (UID: \"0e3d5731-bf9e-4f2c-939b-0706ca41780b\") " pod="openshift-must-gather-5djh2/perf-node-gather-daemonset-x8pmg" Sep 09 21:09:39.659873 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:39.659866 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0e3d5731-bf9e-4f2c-939b-0706ca41780b-podres\") pod \"perf-node-gather-daemonset-x8pmg\" (UID: \"0e3d5731-bf9e-4f2c-939b-0706ca41780b\") " pod="openshift-must-gather-5djh2/perf-node-gather-daemonset-x8pmg" Sep 09 21:09:39.660024 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:39.659918 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e3d5731-bf9e-4f2c-939b-0706ca41780b-lib-modules\") pod \"perf-node-gather-daemonset-x8pmg\" (UID: \"0e3d5731-bf9e-4f2c-939b-0706ca41780b\") " pod="openshift-must-gather-5djh2/perf-node-gather-daemonset-x8pmg" Sep 09 21:09:39.718446 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:39.718415 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-789f7f47d6-tcbm2_68724704-7c68-4fdf-9a76-b20b095de703/console/0.log" Sep 09 21:09:39.760378 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:39.760314 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e3d5731-bf9e-4f2c-939b-0706ca41780b-lib-modules\") pod \"perf-node-gather-daemonset-x8pmg\" (UID: \"0e3d5731-bf9e-4f2c-939b-0706ca41780b\") " pod="openshift-must-gather-5djh2/perf-node-gather-daemonset-x8pmg" Sep 09 21:09:39.760378 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:39.760350 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e3d5731-bf9e-4f2c-939b-0706ca41780b-sys\") pod \"perf-node-gather-daemonset-x8pmg\" (UID: \"0e3d5731-bf9e-4f2c-939b-0706ca41780b\") " pod="openshift-must-gather-5djh2/perf-node-gather-daemonset-x8pmg" Sep 09 21:09:39.760549 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:39.760394 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpq52\" (UniqueName: \"kubernetes.io/projected/0e3d5731-bf9e-4f2c-939b-0706ca41780b-kube-api-access-gpq52\") pod \"perf-node-gather-daemonset-x8pmg\" (UID: \"0e3d5731-bf9e-4f2c-939b-0706ca41780b\") " pod="openshift-must-gather-5djh2/perf-node-gather-daemonset-x8pmg" Sep 09 21:09:39.760549 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:39.760445 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0e3d5731-bf9e-4f2c-939b-0706ca41780b-proc\") pod \"perf-node-gather-daemonset-x8pmg\" (UID: \"0e3d5731-bf9e-4f2c-939b-0706ca41780b\") " pod="openshift-must-gather-5djh2/perf-node-gather-daemonset-x8pmg" Sep 09 21:09:39.760549 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:39.760479 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e3d5731-bf9e-4f2c-939b-0706ca41780b-lib-modules\") pod \"perf-node-gather-daemonset-x8pmg\" (UID: \"0e3d5731-bf9e-4f2c-939b-0706ca41780b\") " pod="openshift-must-gather-5djh2/perf-node-gather-daemonset-x8pmg" Sep 09 21:09:39.760549 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:39.760480 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0e3d5731-bf9e-4f2c-939b-0706ca41780b-podres\") pod \"perf-node-gather-daemonset-x8pmg\" (UID: \"0e3d5731-bf9e-4f2c-939b-0706ca41780b\") " pod="openshift-must-gather-5djh2/perf-node-gather-daemonset-x8pmg" Sep 09 21:09:39.760549 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:39.760504 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e3d5731-bf9e-4f2c-939b-0706ca41780b-sys\") pod \"perf-node-gather-daemonset-x8pmg\" (UID: \"0e3d5731-bf9e-4f2c-939b-0706ca41780b\") " pod="openshift-must-gather-5djh2/perf-node-gather-daemonset-x8pmg" Sep 09 21:09:39.760747 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:39.760552 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0e3d5731-bf9e-4f2c-939b-0706ca41780b-proc\") pod \"perf-node-gather-daemonset-x8pmg\" (UID: \"0e3d5731-bf9e-4f2c-939b-0706ca41780b\") " pod="openshift-must-gather-5djh2/perf-node-gather-daemonset-x8pmg" Sep 09 21:09:39.760747 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:39.760569 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0e3d5731-bf9e-4f2c-939b-0706ca41780b-podres\") pod \"perf-node-gather-daemonset-x8pmg\" (UID: \"0e3d5731-bf9e-4f2c-939b-0706ca41780b\") " pod="openshift-must-gather-5djh2/perf-node-gather-daemonset-x8pmg" Sep 09 21:09:39.770472 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:39.770454 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpq52\" (UniqueName: \"kubernetes.io/projected/0e3d5731-bf9e-4f2c-939b-0706ca41780b-kube-api-access-gpq52\") pod \"perf-node-gather-daemonset-x8pmg\" (UID: \"0e3d5731-bf9e-4f2c-939b-0706ca41780b\") " pod="openshift-must-gather-5djh2/perf-node-gather-daemonset-x8pmg" Sep 09 21:09:39.791382 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:39.791361 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5djh2/perf-node-gather-daemonset-x8pmg" Sep 09 21:09:40.060487 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:40.060456 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5djh2/perf-node-gather-daemonset-x8pmg"] Sep 09 21:09:40.062505 ip-10-0-137-19 kubenswrapper[2580]: W0909 21:09:40.062476 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0e3d5731_bf9e_4f2c_939b_0706ca41780b.slice/crio-1ea55b332dc039c70de75456612dc08b29051284f240913c3438472319184f26 WatchSource:0}: Error finding container 1ea55b332dc039c70de75456612dc08b29051284f240913c3438472319184f26: Status 404 returned error can't find the container with id 1ea55b332dc039c70de75456612dc08b29051284f240913c3438472319184f26 Sep 09 21:09:40.452415 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:40.452383 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5djh2/perf-node-gather-daemonset-x8pmg" event={"ID":"0e3d5731-bf9e-4f2c-939b-0706ca41780b","Type":"ContainerStarted","Data":"5566047447b271bb7a5fe2572b4f838c4ba2e0896c51687b6f8703272f7a5236"} Sep 09 21:09:40.452571 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:40.452434 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5djh2/perf-node-gather-daemonset-x8pmg" event={"ID":"0e3d5731-bf9e-4f2c-939b-0706ca41780b","Type":"ContainerStarted","Data":"1ea55b332dc039c70de75456612dc08b29051284f240913c3438472319184f26"} Sep 09 21:09:40.452571 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:40.452501 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5djh2/perf-node-gather-daemonset-x8pmg" Sep 09 21:09:40.471508 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:40.471469 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5djh2/perf-node-gather-daemonset-x8pmg" podStartSLOduration=1.471457207 podStartE2EDuration="1.471457207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:09:40.46978776 +0000 UTC m=+1157.930429681" watchObservedRunningTime="2026-09-09 21:09:40.471457207 +0000 UTC m=+1157.932099129" Sep 09 21:09:41.684156 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:41.684131 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gb8k5_81bd5151-ce78-4a7c-9ebf-b787d915430c/dns/0.log" Sep 09 21:09:41.707214 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:41.707193 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gb8k5_81bd5151-ce78-4a7c-9ebf-b787d915430c/kube-rbac-proxy/0.log" Sep 09 21:09:41.763036 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:41.763010 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bxfjt_717799e6-8e3b-4a16-88d6-384d950d5385/dns-node-resolver/0.log" Sep 09 21:09:42.359609 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:42.359586 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gx8lm_e64d1b19-5ae3-4758-8753-e83a736855f0/node-ca/0.log" Sep 09 21:09:43.508618 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:43.508590 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rmpg9_4597f764-9265-44e0-b17b-660f72201d59/istio-proxy/1.log" Sep 09 21:09:43.509077 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:43.509059 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rmpg9_4597f764-9265-44e0-b17b-660f72201d59/istio-proxy/0.log" Sep 09 21:09:44.210032 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:44.210010 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-77lm8_bea0cf71-87de-42ff-bcef-683009062999/serve-healthcheck-canary/0.log" Sep 09 21:09:44.742868 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:44.742822 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6pjzl_13b9d045-0f29-42c6-b394-53a075824ec8/kube-rbac-proxy/0.log" Sep 09 21:09:44.766108 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:44.766086 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6pjzl_13b9d045-0f29-42c6-b394-53a075824ec8/exporter/0.log" Sep 09 21:09:44.789602 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:44.789580 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6pjzl_13b9d045-0f29-42c6-b394-53a075824ec8/extractor/0.log" Sep 09 21:09:45.038458 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:45.038401 2580 scope.go:117] "RemoveContainer" containerID="c875dd5c204bdb8440911137eb2a5a41331ecff8dae88e83d1e14ef3314f4729" Sep 09 21:09:45.038595 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:09:45.038576 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=premium-simulated-simulated-premium-kserve-5758cf6949-8chmp_llm(4d7e7307-f923-4b20-9596-9da76d196e94)\"" pod="llm/premium-simulated-simulated-premium-kserve-5758cf6949-8chmp" podUID="4d7e7307-f923-4b20-9596-9da76d196e94" Sep 09 21:09:46.466052 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:46.466026 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5djh2/perf-node-gather-daemonset-x8pmg" Sep 09 21:09:47.120741 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:47.120713 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-m6bbc_64b5633f-b9c9-4d46-9224-fd46d2e5cb01/manager/0.log" Sep 09 21:09:47.197022 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:47.197000 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-njk78_4e582a30-e52b-4a4e-9b26-f4e0f7ec57d4/manager/0.log" Sep 09 21:09:47.262582 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:47.262555 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d8ddb97f8-hzjcq_14ba4026-2fb2-4334-964f-06cc61450890/manager/0.log" Sep 09 21:09:47.593494 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:47.593467 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cf9547dff-vpfp4_a43b1176-2185-44bd-acf9-5b24408e9095/manager/0.log" Sep 09 21:09:47.619532 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:47.619510 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cf9547dff-vpfp4_a43b1176-2185-44bd-acf9-5b24408e9095/copy-manifests/0.log" Sep 09 21:09:49.009098 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:49.009057 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5fc8dd9cff-8tm7j_ad6dd721-249e-499e-b34f-0196649300fa/manager/0.log" Sep 09 21:09:49.038274 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:49.038250 2580 scope.go:117] "RemoveContainer" containerID="e55767363f11486a984eee5494d7a842662273043d124fd83aa81a39a84f82fa" Sep 09 21:09:49.038410 ip-10-0-137-19 kubenswrapper[2580]: E0909 21:09:49.038395 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-78dhn_llm(d957765a-bd0b-4764-9abc-23a64ba4d6d3)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-78dhn" podUID="d957765a-bd0b-4764-9abc-23a64ba4d6d3" Sep 09 21:09:50.061622 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:50.061584 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rmpg9_4597f764-9265-44e0-b17b-660f72201d59/istio-proxy/1.log" Sep 09 21:09:50.062153 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:50.062129 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rmpg9_4597f764-9265-44e0-b17b-660f72201d59/istio-proxy/0.log" Sep 09 21:09:51.388743 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:51.388659 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rmpg9_4597f764-9265-44e0-b17b-660f72201d59/istio-proxy/1.log" Sep 09 21:09:51.389273 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:51.389215 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rmpg9_4597f764-9265-44e0-b17b-660f72201d59/istio-proxy/0.log" Sep 09 21:09:55.582791 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:55.582760 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-95fw4_b8cbb806-0e1f-40f8-9595-3001333438ea/kube-multus/0.log" Sep 09 21:09:55.960563 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:55.960534 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rwtrs_75366d82-a9b6-4a08-a9cf-a3c9cd38ef85/kube-multus-additional-cni-plugins/0.log" Sep 09 21:09:55.986378 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:55.986356 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rwtrs_75366d82-a9b6-4a08-a9cf-a3c9cd38ef85/egress-router-binary-copy/0.log" Sep 09 21:09:56.009108 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:56.009082 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rwtrs_75366d82-a9b6-4a08-a9cf-a3c9cd38ef85/cni-plugins/0.log" Sep 09 21:09:56.031718 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:56.031697 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rwtrs_75366d82-a9b6-4a08-a9cf-a3c9cd38ef85/bond-cni-plugin/0.log" Sep 09 21:09:56.054741 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:56.054721 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rwtrs_75366d82-a9b6-4a08-a9cf-a3c9cd38ef85/routeoverride-cni/0.log" Sep 09 21:09:56.077634 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:56.077611 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rwtrs_75366d82-a9b6-4a08-a9cf-a3c9cd38ef85/whereabouts-cni-bincopy/0.log" Sep 09 21:09:56.103248 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:56.103222 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rwtrs_75366d82-a9b6-4a08-a9cf-a3c9cd38ef85/whereabouts-cni/0.log" Sep 09 21:09:56.262803 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:56.262753 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9dfd9_f8e8b1d2-6523-403e-bace-b96b38d8449b/network-metrics-daemon/0.log" Sep 09 21:09:56.285051 ip-10-0-137-19 kubenswrapper[2580]: I0909 21:09:56.285031 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9dfd9_f8e8b1d2-6523-403e-bace-b96b38d8449b/kube-rbac-proxy/0.log"