Sep 09 22:58:59.115691 ip-10-0-139-252 systemd[1]: Starting Kubernetes Kubelet... Sep 09 22:58:59.619355 ip-10-0-139-252 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 22:58:59.619355 ip-10-0-139-252 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 09 22:58:59.619355 ip-10-0-139-252 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 22:58:59.619355 ip-10-0-139-252 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 09 22:58:59.619355 ip-10-0-139-252 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 22:58:59.619355 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.619054 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 09 22:58:59.625445 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625421 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 22:58:59.625445 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625440 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 22:58:59.625445 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625444 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 22:58:59.625445 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625447 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 22:58:59.625445 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625450 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 22:58:59.625445 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625453 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 22:58:59.625445 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625456 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 22:58:59.625726 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625459 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 22:58:59.625726 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625463 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 22:58:59.625726 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625465 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 22:58:59.625726 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625468 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 22:58:59.625726 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625471 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 22:58:59.625726 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625475 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 22:58:59.625726 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625480 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 22:58:59.625726 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625483 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 22:58:59.625726 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625486 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 22:58:59.625726 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625489 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 22:58:59.625726 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625492 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 22:58:59.625726 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625495 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 22:58:59.625726 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625497 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 22:58:59.625726 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625500 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 22:58:59.625726 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625502 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 22:58:59.625726 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625505 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 22:58:59.625726 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625508 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 22:58:59.625726 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625510 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 22:58:59.625726 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625513 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 22:58:59.625726 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625515 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 22:58:59.626207 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625517 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 22:58:59.626207 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625520 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 22:58:59.626207 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625523 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 22:58:59.626207 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625525 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 22:58:59.626207 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625528 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 22:58:59.626207 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625530 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 22:58:59.626207 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625533 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 22:58:59.626207 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625535 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 22:58:59.626207 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625538 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 22:58:59.626207 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625540 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 22:58:59.626207 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625543 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 22:58:59.626207 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625545 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 22:58:59.626207 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625548 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 22:58:59.626207 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625551 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 22:58:59.626207 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625554 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 22:58:59.626207 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625556 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 22:58:59.626207 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625559 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 22:58:59.626207 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625562 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 22:58:59.626207 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625564 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 22:58:59.626695 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625566 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 22:58:59.626695 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625569 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 22:58:59.626695 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625574 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 22:58:59.626695 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625578 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 22:58:59.626695 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625583 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 22:58:59.626695 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625585 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 22:58:59.626695 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625588 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 22:58:59.626695 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625591 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 09 22:58:59.626695 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625593 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 22:58:59.626695 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625596 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 22:58:59.626695 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625598 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 22:58:59.626695 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625601 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 22:58:59.626695 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625603 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 22:58:59.626695 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625606 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 22:58:59.626695 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625608 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 22:58:59.626695 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625611 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 22:58:59.626695 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625613 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 22:58:59.626695 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625616 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 22:58:59.626695 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625618 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 22:58:59.626695 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625620 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 22:58:59.627175 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625623 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 22:58:59.627175 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625626 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 22:58:59.627175 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625628 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 22:58:59.627175 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625631 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 22:58:59.627175 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625633 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 22:58:59.627175 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625636 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 22:58:59.627175 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625642 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 22:58:59.627175 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625644 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 22:58:59.627175 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625647 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 22:58:59.627175 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625650 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 22:58:59.627175 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625654 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 22:58:59.627175 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625658 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 22:58:59.627175 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625661 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 22:58:59.627175 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625664 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 22:58:59.627175 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625667 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 22:58:59.627175 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625670 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 22:58:59.627175 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625673 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 22:58:59.627175 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625676 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 22:58:59.627175 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625679 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 22:58:59.627768 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625681 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 22:58:59.627768 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625684 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 22:58:59.627768 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625686 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 22:58:59.627768 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625689 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 22:58:59.627768 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625692 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 22:58:59.627768 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625695 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 22:58:59.627768 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625697 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 22:58:59.627768 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625700 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 22:58:59.627768 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625702 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 22:58:59.627768 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625705 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 22:58:59.627768 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625707 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 22:58:59.627768 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625710 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 22:58:59.627768 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625712 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 22:58:59.627768 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.625715 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 22:58:59.627768 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626442 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 22:58:59.627768 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626449 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 22:58:59.627768 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626452 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 22:58:59.627768 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626455 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 22:58:59.627768 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626457 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 22:58:59.628240 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626461 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 22:58:59.628240 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626464 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 22:58:59.628240 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626466 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 22:58:59.628240 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626469 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 22:58:59.628240 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626471 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 22:58:59.628240 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626474 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 22:58:59.628240 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626476 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 22:58:59.628240 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626479 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 22:58:59.628240 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626482 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 22:58:59.628240 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626484 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 22:58:59.628240 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626486 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 22:58:59.628240 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626491 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 22:58:59.628240 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626496 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 22:58:59.628240 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626499 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 22:58:59.628240 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626502 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 22:58:59.628240 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626505 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 22:58:59.628240 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626508 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 22:58:59.628240 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626510 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 22:58:59.628240 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626513 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 22:58:59.628772 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626516 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 22:58:59.628772 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626519 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 22:58:59.628772 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626521 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 22:58:59.628772 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626524 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 22:58:59.628772 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626529 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 22:58:59.628772 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626532 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 22:58:59.628772 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626535 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 22:58:59.628772 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626537 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 22:58:59.628772 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626540 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 22:58:59.628772 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626543 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 22:58:59.628772 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626546 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 22:58:59.628772 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626548 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 22:58:59.628772 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626551 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 22:58:59.628772 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626554 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 22:58:59.628772 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626556 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 22:58:59.628772 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626560 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 22:58:59.628772 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626564 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 22:58:59.628772 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626567 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 22:58:59.628772 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626569 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 22:58:59.629245 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626572 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 22:58:59.629245 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626575 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 22:58:59.629245 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626577 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 22:58:59.629245 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626580 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 22:58:59.629245 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626583 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 22:58:59.629245 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626585 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 22:58:59.629245 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626588 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 22:58:59.629245 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626593 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 22:58:59.629245 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626595 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 22:58:59.629245 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626598 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 22:58:59.629245 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626601 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 22:58:59.629245 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626603 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 22:58:59.629245 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626606 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 22:58:59.629245 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626608 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 22:58:59.629245 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626611 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 22:58:59.629245 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626613 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 22:58:59.629245 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626616 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 22:58:59.629245 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626619 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 22:58:59.629245 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626621 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 22:58:59.629245 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626624 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 22:58:59.629765 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626626 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 22:58:59.629765 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626628 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 22:58:59.629765 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626631 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 22:58:59.629765 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626633 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 22:58:59.629765 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626636 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 22:58:59.629765 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626638 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 22:58:59.629765 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626641 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 22:58:59.629765 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626644 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 22:58:59.629765 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626646 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 22:58:59.629765 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626648 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 22:58:59.629765 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626651 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 22:58:59.629765 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626654 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 22:58:59.629765 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626656 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 22:58:59.629765 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626659 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 22:58:59.629765 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626661 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 22:58:59.629765 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626663 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 22:58:59.629765 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626666 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 22:58:59.629765 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626668 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 22:58:59.629765 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626670 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 22:58:59.629765 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626673 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 22:58:59.630264 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626676 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 22:58:59.630264 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626678 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 22:58:59.630264 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626681 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 22:58:59.630264 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626683 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 22:58:59.630264 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626685 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 22:58:59.630264 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626688 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 09 22:58:59.630264 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626690 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 22:58:59.630264 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626692 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 22:58:59.630264 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626695 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 22:58:59.630264 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626698 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 22:58:59.630264 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626700 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 22:58:59.630264 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626703 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 22:58:59.630264 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626706 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 22:58:59.630264 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626708 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 22:58:59.630264 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626711 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 22:58:59.630264 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.626713 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 22:58:59.630264 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627102 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 22:58:59.630264 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627107 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 22:58:59.630264 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627110 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 22:58:59.630264 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627113 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 22:58:59.630264 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627115 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 22:58:59.630795 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627118 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 22:58:59.630795 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627120 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 22:58:59.630795 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627123 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 22:58:59.630795 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627125 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 22:58:59.630795 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627128 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 22:58:59.630795 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627131 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 22:58:59.630795 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627134 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 22:58:59.630795 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627136 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 22:58:59.630795 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627139 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 22:58:59.630795 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627141 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 22:58:59.630795 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627144 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 22:58:59.630795 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627148 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 22:58:59.630795 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627150 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 22:58:59.630795 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627153 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 22:58:59.630795 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627155 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 22:58:59.630795 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627158 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 22:58:59.630795 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627160 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 22:58:59.630795 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627163 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 22:58:59.630795 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627165 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 22:58:59.630795 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627168 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 22:58:59.631284 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627171 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 22:58:59.631284 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627173 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 22:58:59.631284 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627175 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 22:58:59.631284 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627178 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 22:58:59.631284 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627180 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 22:58:59.631284 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627183 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 22:58:59.631284 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627185 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 22:58:59.631284 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627190 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 22:58:59.631284 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627193 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 22:58:59.631284 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627197 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 22:58:59.631284 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627199 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 22:58:59.631284 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627202 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 22:58:59.631284 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627204 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 22:58:59.631284 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627207 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 22:58:59.631284 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627209 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 22:58:59.631284 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627211 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 22:58:59.631284 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627214 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 22:58:59.631284 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627216 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 22:58:59.631284 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627219 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 22:58:59.631284 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627221 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 22:58:59.631788 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627223 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 22:58:59.631788 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627226 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 22:58:59.631788 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627228 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 22:58:59.631788 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627231 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 22:58:59.631788 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627234 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 22:58:59.631788 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627236 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 22:58:59.631788 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627239 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 22:58:59.631788 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627241 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 22:58:59.631788 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627244 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 22:58:59.631788 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627246 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 22:58:59.631788 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627248 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 22:58:59.631788 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627251 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 22:58:59.631788 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627253 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 22:58:59.631788 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627256 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 22:58:59.631788 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627259 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 22:58:59.631788 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627261 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 22:58:59.631788 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627264 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 09 22:58:59.631788 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627266 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 22:58:59.631788 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627270 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 22:58:59.632266 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627275 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 22:58:59.632266 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627278 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 22:58:59.632266 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627281 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 22:58:59.632266 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627283 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 22:58:59.632266 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627286 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 22:58:59.632266 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627288 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 22:58:59.632266 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627291 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 22:58:59.632266 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627293 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 22:58:59.632266 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627296 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 22:58:59.632266 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627298 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 22:58:59.632266 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627300 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 22:58:59.632266 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627303 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 22:58:59.632266 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627305 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 22:58:59.632266 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627310 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 22:58:59.632266 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627313 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 22:58:59.632266 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627316 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 22:58:59.632266 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627319 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 22:58:59.632266 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627321 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 22:58:59.632266 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627324 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 22:58:59.632758 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627327 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 22:58:59.632758 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627330 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 22:58:59.632758 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627332 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 22:58:59.632758 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627335 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 22:58:59.632758 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627337 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 22:58:59.632758 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627343 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 22:58:59.632758 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627346 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 22:58:59.632758 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627349 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 22:58:59.632758 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627351 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 22:58:59.632758 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627354 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 22:58:59.632758 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627356 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 22:58:59.632758 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627359 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 22:58:59.632758 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627361 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 22:58:59.632758 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627366 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 22:58:59.632758 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627368 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 22:58:59.632758 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.627371 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 22:58:59.632758 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627466 2578 flags.go:64] FLAG: --address="0.0.0.0" Sep 09 22:58:59.632758 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627500 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 09 22:58:59.632758 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627506 2578 flags.go:64] FLAG: --anonymous-auth="true" Sep 09 22:58:59.632758 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627511 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 09 22:58:59.632758 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627515 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 09 22:58:59.633265 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627519 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 09 22:58:59.633265 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627523 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 09 22:58:59.633265 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627528 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 09 22:58:59.633265 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627531 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 09 22:58:59.633265 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627534 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 09 22:58:59.633265 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627537 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 09 22:58:59.633265 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627541 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 09 22:58:59.633265 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627544 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 09 22:58:59.633265 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627547 2578 flags.go:64] FLAG: --cgroup-root="" Sep 09 22:58:59.633265 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627549 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 09 22:58:59.633265 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627552 2578 flags.go:64] FLAG: --client-ca-file="" Sep 09 22:58:59.633265 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627556 2578 flags.go:64] FLAG: --cloud-provider="external" Sep 09 22:58:59.633265 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627560 2578 flags.go:64] FLAG: --cluster-dns="[]" Sep 09 22:58:59.633265 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627564 2578 flags.go:64] FLAG: --cluster-domain="" Sep 09 22:58:59.633265 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627567 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 09 22:58:59.633265 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627570 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 09 22:58:59.633265 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627574 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 09 22:58:59.633265 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627578 2578 flags.go:64] FLAG: --container-log-max-files="5" Sep 09 22:58:59.633265 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627582 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 09 22:58:59.633265 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627584 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 09 22:58:59.633265 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627587 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 09 22:58:59.633265 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627591 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 09 22:58:59.633265 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627594 2578 flags.go:64] FLAG: --contention-profiling="false" Sep 09 22:58:59.633265 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627597 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 09 22:58:59.633904 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627601 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 09 22:58:59.633904 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627604 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 09 22:58:59.633904 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627607 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 09 22:58:59.633904 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627611 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 09 22:58:59.633904 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627614 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 09 22:58:59.633904 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627617 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 09 22:58:59.633904 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627620 2578 flags.go:64] FLAG: --enable-load-reader="false" Sep 09 22:58:59.633904 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627622 2578 flags.go:64] FLAG: --enable-server="true" Sep 09 22:58:59.633904 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627625 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 09 22:58:59.633904 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627629 2578 flags.go:64] FLAG: --event-burst="100" Sep 09 22:58:59.633904 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627632 2578 flags.go:64] FLAG: --event-qps="50" Sep 09 22:58:59.633904 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627635 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 09 22:58:59.633904 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627638 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 09 22:58:59.633904 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627641 2578 flags.go:64] FLAG: --eviction-hard="" Sep 09 22:58:59.633904 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627645 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 09 22:58:59.633904 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627648 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 09 22:58:59.633904 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627651 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 09 22:58:59.633904 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627654 2578 flags.go:64] FLAG: --eviction-soft="" Sep 09 22:58:59.633904 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627656 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 09 22:58:59.633904 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627659 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 09 22:58:59.633904 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627662 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 09 22:58:59.633904 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627665 2578 flags.go:64] FLAG: --experimental-mounter-path="" Sep 09 22:58:59.633904 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627668 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 09 22:58:59.633904 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627671 2578 flags.go:64] FLAG: --fail-swap-on="true" Sep 09 22:58:59.633904 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627675 2578 flags.go:64] FLAG: --feature-gates="" Sep 09 22:58:59.634544 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627679 2578 flags.go:64] FLAG: --file-check-frequency="20s" Sep 09 22:58:59.634544 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627682 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 09 22:58:59.634544 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627684 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 09 22:58:59.634544 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627687 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 09 22:58:59.634544 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627690 2578 flags.go:64] FLAG: --healthz-port="10248" Sep 09 22:58:59.634544 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627693 2578 flags.go:64] FLAG: --help="false" Sep 09 22:58:59.634544 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627696 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-139-252.ec2.internal" Sep 09 22:58:59.634544 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627700 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 09 22:58:59.634544 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627703 2578 flags.go:64] FLAG: --http-check-frequency="20s" Sep 09 22:58:59.634544 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627706 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 09 22:58:59.634544 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627709 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 09 22:58:59.634544 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627713 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 09 22:58:59.634544 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627715 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 09 22:58:59.634544 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627718 2578 flags.go:64] FLAG: --image-service-endpoint="" Sep 09 22:58:59.634544 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627721 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 09 22:58:59.634544 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627724 2578 flags.go:64] FLAG: --kube-api-burst="100" Sep 09 22:58:59.634544 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627727 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 09 22:58:59.634544 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627730 2578 flags.go:64] FLAG: --kube-api-qps="50" Sep 09 22:58:59.634544 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627732 2578 flags.go:64] FLAG: --kube-reserved="" Sep 09 22:58:59.634544 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627735 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 09 22:58:59.634544 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627738 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 09 22:58:59.634544 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627741 2578 flags.go:64] FLAG: --kubelet-cgroups="" Sep 09 22:58:59.634544 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627744 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 09 22:58:59.634544 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627747 2578 flags.go:64] FLAG: --lock-file="" Sep 09 22:58:59.635138 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627749 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 09 22:58:59.635138 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627752 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 09 22:58:59.635138 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627755 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 09 22:58:59.635138 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627760 2578 flags.go:64] FLAG: --log-json-split-stream="false" Sep 09 22:58:59.635138 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627764 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 09 22:58:59.635138 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627767 2578 flags.go:64] FLAG: --log-text-split-stream="false" Sep 09 22:58:59.635138 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627770 2578 flags.go:64] FLAG: --logging-format="text" Sep 09 22:58:59.635138 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627774 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 09 22:58:59.635138 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627777 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 09 22:58:59.635138 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627780 2578 flags.go:64] FLAG: --manifest-url="" Sep 09 22:58:59.635138 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627783 2578 flags.go:64] FLAG: --manifest-url-header="" Sep 09 22:58:59.635138 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627787 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 09 22:58:59.635138 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627790 2578 flags.go:64] FLAG: --max-open-files="1000000" Sep 09 22:58:59.635138 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627794 2578 flags.go:64] FLAG: --max-pods="110" Sep 09 22:58:59.635138 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627797 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 09 22:58:59.635138 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627801 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 09 22:58:59.635138 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627804 2578 flags.go:64] FLAG: --memory-manager-policy="None" Sep 09 22:58:59.635138 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627806 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 09 22:58:59.635138 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627809 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 09 22:58:59.635138 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627812 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 09 22:58:59.635138 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627815 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 09 22:58:59.635138 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627823 2578 flags.go:64] FLAG: --node-status-max-images="50" Sep 09 22:58:59.635138 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627826 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 09 22:58:59.635138 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627829 2578 flags.go:64] FLAG: --oom-score-adj="-999" Sep 09 22:58:59.635138 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627832 2578 flags.go:64] FLAG: --pod-cidr="" Sep 09 22:58:59.636054 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627835 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 09 22:58:59.636054 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627840 2578 flags.go:64] FLAG: --pod-manifest-path="" Sep 09 22:58:59.636054 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627842 2578 flags.go:64] FLAG: --pod-max-pids="-1" Sep 09 22:58:59.636054 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627845 2578 flags.go:64] FLAG: --pods-per-core="0" Sep 09 22:58:59.636054 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627848 2578 flags.go:64] FLAG: --port="10250" Sep 09 22:58:59.636054 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627851 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 09 22:58:59.636054 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627854 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07c47be34b88d631d" Sep 09 22:58:59.636054 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627857 2578 flags.go:64] FLAG: --qos-reserved="" Sep 09 22:58:59.636054 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627860 2578 flags.go:64] FLAG: --read-only-port="10255" Sep 09 22:58:59.636054 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627863 2578 flags.go:64] FLAG: --register-node="true" Sep 09 22:58:59.636054 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627865 2578 flags.go:64] FLAG: --register-with-taints="" Sep 09 22:58:59.636054 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627869 2578 flags.go:64] FLAG: --registry-burst="10" Sep 09 22:58:59.636054 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627872 2578 flags.go:64] FLAG: --registry-qps="5" Sep 09 22:58:59.636054 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627875 2578 flags.go:64] FLAG: --reserved-cpus="" Sep 09 22:58:59.636054 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627878 2578 flags.go:64] FLAG: --reserved-memory="" Sep 09 22:58:59.636054 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627882 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 09 22:58:59.636054 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627885 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 09 22:58:59.636054 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627889 2578 flags.go:64] FLAG: --rotate-certificates="false" Sep 09 22:58:59.636054 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627892 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 09 22:58:59.636054 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627894 2578 flags.go:64] FLAG: --runonce="false" Sep 09 22:58:59.636054 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627897 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 09 22:58:59.636054 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627900 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 09 22:58:59.636054 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627904 2578 flags.go:64] FLAG: --seccomp-default="false" Sep 09 22:58:59.636054 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627907 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 09 22:58:59.636054 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627910 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 09 22:58:59.637120 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627913 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 09 22:58:59.637120 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627916 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 09 22:58:59.637120 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627919 2578 flags.go:64] FLAG: --storage-driver-password="root" Sep 09 22:58:59.637120 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627922 2578 flags.go:64] FLAG: --storage-driver-secure="false" Sep 09 22:58:59.637120 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627925 2578 flags.go:64] FLAG: --storage-driver-table="stats" Sep 09 22:58:59.637120 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627927 2578 flags.go:64] FLAG: --storage-driver-user="root" Sep 09 22:58:59.637120 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627930 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 09 22:58:59.637120 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627933 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 09 22:58:59.637120 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627936 2578 flags.go:64] FLAG: --system-cgroups="" Sep 09 22:58:59.637120 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627938 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 09 22:58:59.637120 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627967 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 09 22:58:59.637120 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627970 2578 flags.go:64] FLAG: --tls-cert-file="" Sep 09 22:58:59.637120 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627973 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 09 22:58:59.637120 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627978 2578 flags.go:64] FLAG: --tls-min-version="" Sep 09 22:58:59.637120 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627980 2578 flags.go:64] FLAG: --tls-private-key-file="" Sep 09 22:58:59.637120 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627983 2578 flags.go:64] FLAG: --topology-manager-policy="none" Sep 09 22:58:59.637120 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627986 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 09 22:58:59.637120 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627989 2578 flags.go:64] FLAG: --topology-manager-scope="container" Sep 09 22:58:59.637120 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627992 2578 flags.go:64] FLAG: --v="2" Sep 09 22:58:59.637120 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.627996 2578 flags.go:64] FLAG: --version="false" Sep 09 22:58:59.637120 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.628000 2578 flags.go:64] FLAG: --vmodule="" Sep 09 22:58:59.637120 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.628004 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 09 22:58:59.637120 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.628010 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 09 22:58:59.637120 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628131 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 22:58:59.637120 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628135 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 22:58:59.638196 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628139 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 22:58:59.638196 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628141 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 22:58:59.638196 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628144 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 22:58:59.638196 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628158 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 22:58:59.638196 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628164 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 22:58:59.638196 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628167 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 22:58:59.638196 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628170 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 22:58:59.638196 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628173 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 22:58:59.638196 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628175 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 22:58:59.638196 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628178 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 22:58:59.638196 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628180 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 22:58:59.638196 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628183 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 22:58:59.638196 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628185 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 22:58:59.638196 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628188 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 22:58:59.638196 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628191 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 22:58:59.638196 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628193 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 22:58:59.638196 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628196 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 22:58:59.638196 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628199 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 22:58:59.638196 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628201 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 22:58:59.639087 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628204 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 22:58:59.639087 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628206 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 22:58:59.639087 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628209 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 22:58:59.639087 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628211 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 22:58:59.639087 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628213 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 22:58:59.639087 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628216 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 22:58:59.639087 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628218 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 22:58:59.639087 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628221 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 22:58:59.639087 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628223 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 22:58:59.639087 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628226 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 22:58:59.639087 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628230 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 22:58:59.639087 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628232 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 22:58:59.639087 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628235 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 22:58:59.639087 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628237 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 22:58:59.639087 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628240 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 22:58:59.639087 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628243 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 22:58:59.639087 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628246 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 22:58:59.639087 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628249 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 22:58:59.639087 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628252 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 22:58:59.639660 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628255 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 22:58:59.639660 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628257 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 22:58:59.639660 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628260 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 22:58:59.639660 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628262 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 22:58:59.639660 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628265 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 22:58:59.639660 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628267 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 22:58:59.639660 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628269 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 22:58:59.639660 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628273 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 22:58:59.639660 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628277 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 22:58:59.639660 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628280 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 22:58:59.639660 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628282 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 22:58:59.639660 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628285 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 22:58:59.639660 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628288 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 22:58:59.639660 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628291 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 22:58:59.639660 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628293 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 22:58:59.639660 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628296 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 22:58:59.639660 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628299 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 22:58:59.639660 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628301 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 22:58:59.639660 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628304 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 22:58:59.640506 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628306 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 22:58:59.640506 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628309 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 22:58:59.640506 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628311 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 22:58:59.640506 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628314 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 22:58:59.640506 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628318 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 22:58:59.640506 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628320 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 22:58:59.640506 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628323 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 22:58:59.640506 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628325 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 22:58:59.640506 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628327 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 22:58:59.640506 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628330 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 22:58:59.640506 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628332 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 22:58:59.640506 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628335 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 22:58:59.640506 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628338 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 22:58:59.640506 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628340 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 22:58:59.640506 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628343 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 22:58:59.640506 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628345 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 22:58:59.640506 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628348 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 22:58:59.640506 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628350 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 22:58:59.640506 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628353 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 22:58:59.640506 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628355 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 22:58:59.641322 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628358 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 22:58:59.641322 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628361 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 22:58:59.641322 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628365 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 22:58:59.641322 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628368 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 22:58:59.641322 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628370 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 22:58:59.641322 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628373 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 22:58:59.641322 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628395 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 22:58:59.641322 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628398 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 09 22:58:59.641322 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628401 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 22:58:59.641322 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628403 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 22:58:59.641322 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628406 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 22:58:59.641322 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628409 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 22:58:59.641322 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628411 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 22:58:59.641322 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628414 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 22:58:59.641322 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628416 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 22:58:59.641322 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628418 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 22:58:59.641322 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628422 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 22:58:59.641322 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628425 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 22:58:59.641322 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628428 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 22:58:59.641322 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.628431 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 22:58:59.641862 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.629259 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 22:58:59.641862 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.637744 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 09 22:58:59.641862 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.637768 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 09 22:58:59.641862 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.637791 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 09 22:58:59.641862 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.637806 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 09 22:58:59.641862 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.637889 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 22:58:59.641862 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.637899 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 22:58:59.641862 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.637905 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 22:58:59.641862 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.637910 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 22:58:59.641862 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.637915 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 22:58:59.641862 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.637920 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 22:58:59.641862 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.637924 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 22:58:59.641862 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.637929 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 22:58:59.641862 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.637934 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 22:58:59.642345 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.637940 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 22:58:59.642345 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.637946 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 22:58:59.642345 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.637951 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 22:58:59.642345 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.637956 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 22:58:59.642345 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.637960 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 22:58:59.642345 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.637964 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 22:58:59.642345 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.637968 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 22:58:59.642345 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.637974 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 22:58:59.642345 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.637980 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 22:58:59.642345 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.637985 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 22:58:59.642345 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.637989 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 22:58:59.642345 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.637993 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 22:58:59.642345 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.637997 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 22:58:59.642345 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638002 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 22:58:59.642345 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638006 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 22:58:59.642345 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638010 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 09 22:58:59.642345 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638014 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 22:58:59.642345 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638018 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 22:58:59.642345 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638022 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 22:58:59.642345 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638027 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 22:58:59.642858 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638033 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 22:58:59.642858 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638037 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 22:58:59.642858 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638042 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 22:58:59.642858 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638045 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 22:58:59.642858 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638050 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 22:58:59.642858 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638054 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 22:58:59.642858 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638058 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 22:58:59.642858 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638062 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 22:58:59.642858 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638065 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 22:58:59.642858 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638070 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 22:58:59.642858 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638073 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 22:58:59.642858 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638077 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 22:58:59.642858 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638082 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 22:58:59.642858 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638086 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 22:58:59.642858 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638090 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 22:58:59.642858 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638094 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 22:58:59.642858 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638098 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 22:58:59.642858 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638104 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 22:58:59.642858 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638108 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 22:58:59.642858 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638112 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 22:58:59.643454 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638116 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 22:58:59.643454 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638120 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 22:58:59.643454 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638124 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 22:58:59.643454 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638128 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 22:58:59.643454 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638132 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 22:58:59.643454 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638136 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 22:58:59.643454 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638140 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 22:58:59.643454 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638145 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 22:58:59.643454 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638149 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 22:58:59.643454 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638153 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 22:58:59.643454 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638157 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 22:58:59.643454 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638162 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 22:58:59.643454 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638167 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 22:58:59.643454 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638172 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 22:58:59.643454 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638176 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 22:58:59.643454 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638180 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 22:58:59.643454 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638184 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 22:58:59.643454 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638188 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 22:58:59.643454 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638192 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 22:58:59.643454 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638196 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 22:58:59.643964 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638201 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 22:58:59.643964 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638204 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 22:58:59.643964 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638208 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 22:58:59.643964 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638212 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 22:58:59.643964 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638216 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 22:58:59.643964 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638219 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 22:58:59.643964 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638223 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 22:58:59.643964 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638227 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 22:58:59.643964 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638231 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 22:58:59.643964 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638235 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 22:58:59.643964 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638240 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 22:58:59.643964 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638244 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 22:58:59.643964 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638248 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 22:58:59.643964 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638252 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 22:58:59.643964 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638256 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 22:58:59.643964 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638260 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 22:58:59.643964 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638264 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 22:58:59.643964 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638268 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 22:58:59.643964 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638272 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 22:58:59.643964 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638277 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 22:58:59.644521 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638281 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 22:58:59.644521 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638285 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 22:58:59.644521 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638289 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 22:58:59.644521 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638294 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 22:58:59.644521 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638298 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 22:58:59.644521 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638302 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 22:58:59.644521 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638307 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 22:58:59.644521 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638311 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 22:58:59.644521 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638315 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 22:58:59.644521 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638320 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 22:58:59.644521 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.638328 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 22:58:59.644521 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638530 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 22:58:59.644521 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638538 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 22:58:59.644521 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638544 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 22:58:59.644872 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638551 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 22:58:59.644872 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638555 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 22:58:59.644872 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638559 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 22:58:59.644872 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638564 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 22:58:59.644872 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638568 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 22:58:59.644872 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638572 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 22:58:59.644872 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638576 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 09 22:58:59.644872 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638580 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 22:58:59.644872 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638584 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 22:58:59.644872 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638588 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 22:58:59.644872 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638592 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 22:58:59.644872 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638596 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 22:58:59.644872 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638600 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 22:58:59.644872 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638604 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 22:58:59.644872 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638609 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 22:58:59.644872 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638613 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 22:58:59.644872 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638617 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 22:58:59.644872 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638621 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 22:58:59.644872 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638625 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 22:58:59.644872 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638629 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 22:58:59.645505 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638633 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 22:58:59.645505 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638638 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 22:58:59.645505 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638642 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 22:58:59.645505 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638646 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 22:58:59.645505 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638650 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 22:58:59.645505 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638655 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 22:58:59.645505 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638659 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 22:58:59.645505 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638663 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 22:58:59.645505 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638667 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 22:58:59.645505 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638671 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 22:58:59.645505 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638675 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 22:58:59.645505 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638680 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 22:58:59.645505 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638685 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 22:58:59.645505 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638690 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 22:58:59.645505 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638694 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 22:58:59.645505 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638698 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 22:58:59.645505 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638705 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 22:58:59.645505 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638711 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 22:58:59.645505 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638715 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 22:58:59.645968 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638720 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 22:58:59.645968 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638725 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 22:58:59.645968 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638729 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 22:58:59.645968 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638733 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 22:58:59.645968 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638738 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 22:58:59.645968 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638742 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 22:58:59.645968 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638746 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 22:58:59.645968 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638752 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 22:58:59.645968 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638758 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 22:58:59.645968 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638762 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 22:58:59.645968 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638767 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 22:58:59.645968 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638772 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 22:58:59.645968 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638776 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 22:58:59.645968 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638781 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 22:58:59.645968 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638785 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 22:58:59.645968 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638790 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 22:58:59.645968 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638795 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 22:58:59.645968 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638799 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 22:58:59.645968 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638803 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 22:58:59.646440 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638808 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 22:58:59.646440 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638812 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 22:58:59.646440 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638816 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 22:58:59.646440 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638820 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 22:58:59.646440 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638824 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 22:58:59.646440 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638828 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 22:58:59.646440 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638832 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 22:58:59.646440 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638836 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 22:58:59.646440 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638840 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 22:58:59.646440 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638844 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 22:58:59.646440 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638848 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 22:58:59.646440 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638852 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 22:58:59.646440 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638856 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 22:58:59.646440 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638860 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 22:58:59.646440 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638864 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 22:58:59.646440 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638868 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 22:58:59.646440 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638872 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 22:58:59.646440 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638876 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 22:58:59.646440 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638880 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 22:58:59.646440 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638884 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 22:58:59.647235 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638888 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 22:58:59.647235 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638892 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 22:58:59.647235 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638896 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 22:58:59.647235 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638900 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 22:58:59.647235 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638904 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 22:58:59.647235 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638908 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 22:58:59.647235 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638912 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 22:58:59.647235 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638917 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 22:58:59.647235 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638921 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 22:58:59.647235 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638925 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 22:58:59.647235 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638929 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 22:58:59.647235 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638933 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 22:58:59.647235 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638937 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 22:58:59.647235 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638942 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 22:58:59.647235 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638946 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 22:58:59.647235 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638950 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 22:58:59.647235 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638954 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 22:58:59.647235 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:58:59.638958 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 22:58:59.647881 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.638966 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 22:58:59.647881 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.639136 2578 server.go:962] "Client rotation is on, will bootstrap in background" Sep 09 22:58:59.647881 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.641921 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 09 22:58:59.647881 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.643046 2578 server.go:1019] "Starting client certificate rotation" Sep 09 22:58:59.647881 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.643150 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 22:58:59.647881 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.643179 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 22:58:59.647881 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.647173 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 22:58:59.650561 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.650538 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 22:58:59.668416 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.668390 2578 log.go:25] "Validated CRI v1 runtime API" Sep 09 22:58:59.672684 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.672664 2578 log.go:25] "Validated CRI v1 image API" Sep 09 22:58:59.673554 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.673537 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 09 22:58:59.677097 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.677077 2578 fs.go:135] Filesystem UUIDs: map[2c5a2fff-da2f-4898-8dac-2686d7403ed3:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9df65699-0131-44ba-bad3-46d78a6be79e:/dev/nvme0n1p4] Sep 09 22:58:59.677171 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.677095 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 09 22:58:59.678220 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.678196 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 22:58:59.684103 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.683991 2578 manager.go:216] Machine: {Timestamp:2026-09-09 22:58:59.68207154 +0000 UTC m=+0.432656459 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099443 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fca164956e1b0264d7c1284cd0c06 SystemUUID:ec2fca16-4956-e1b0-264d-7c1284cd0c06 BootID:d4f199e3-f60f-450c-bb7e-910c83b9b29d Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ae:22:23:bd:0b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ae:22:23:bd:0b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:e6:c5:9e:06:10 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 09 22:58:59.684103 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.684102 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 09 22:58:59.684220 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.684196 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 09 22:58:59.688866 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.688838 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 09 22:58:59.689018 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.688871 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-252.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 09 22:58:59.690271 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.690261 2578 topology_manager.go:138] "Creating topology manager with none policy" Sep 09 22:58:59.690304 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.690275 2578 container_manager_linux.go:309] "Creating device plugin manager" Sep 09 22:58:59.690304 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.690283 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 22:58:59.691024 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.691014 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 22:58:59.693692 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.693680 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 09 22:58:59.698359 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.698345 2578 state_mem.go:36] "Initialized new in-memory state store" Sep 09 22:58:59.698494 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.698484 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 09 22:58:59.698530 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.698517 2578 kubelet.go:486] "Attempting to sync node with API server" Sep 09 22:58:59.698566 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.698532 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 09 22:58:59.698566 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.698546 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 09 22:58:59.698628 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.698570 2578 kubelet.go:398] "Adding apiserver pod source" Sep 09 22:58:59.698628 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.698583 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 09 22:58:59.698697 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.698670 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ngp9f" Sep 09 22:58:59.699029 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.699014 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 22:58:59.699100 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.699036 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 22:58:59.701723 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.701705 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 09 22:58:59.704065 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.704042 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ngp9f" Sep 09 22:58:59.705147 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.705133 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 09 22:58:59.705189 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.705164 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 09 22:58:59.705338 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.705327 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 09 22:58:59.705370 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.705343 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 09 22:58:59.705370 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.705349 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 09 22:58:59.705370 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.705355 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 09 22:58:59.705370 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.705362 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 09 22:58:59.705370 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.705371 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 09 22:58:59.706521 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.706508 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 09 22:58:59.706567 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.706524 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 09 22:58:59.706567 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.706535 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 09 22:58:59.706567 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.706543 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 09 22:58:59.706567 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.706556 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 09 22:58:59.706567 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.706567 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 09 22:58:59.708321 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:58:59.708302 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-252.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 09 22:58:59.708392 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:58:59.708302 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 09 22:58:59.708794 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.708782 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 09 22:58:59.708827 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.708797 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 09 22:58:59.711805 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.711781 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-252.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 09 22:58:59.713243 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.713230 2578 server.go:1268] "Started kubelet" Sep 09 22:58:59.713335 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.713310 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 09 22:58:59.713456 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.713389 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 09 22:58:59.713456 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.713430 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 09 22:58:59.714112 ip-10-0-139-252 systemd[1]: Started Kubernetes Kubelet. Sep 09 22:58:59.714958 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.714938 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 09 22:58:59.723628 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:58:59.723605 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 09 22:58:59.723966 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.723946 2578 server.go:310] "Adding debug handlers to kubelet server" Sep 09 22:58:59.727462 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.727444 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 09 22:58:59.727462 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.727453 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 09 22:58:59.728107 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:58:59.727708 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-252.ec2.internal\" not found" Sep 09 22:58:59.728211 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.728155 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 09 22:58:59.728211 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.728168 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 09 22:58:59.728211 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.728179 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 09 22:58:59.728541 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.728237 2578 reconstruct.go:97] "Volume reconstruction finished" Sep 09 22:58:59.728541 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.728257 2578 reconciler.go:29] "Reconciler: start to sync state" Sep 09 22:58:59.728541 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.728311 2578 factory.go:55] Registering systemd factory Sep 09 22:58:59.728541 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.728326 2578 factory.go:223] Registration of the systemd container factory successfully Sep 09 22:58:59.728703 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.728571 2578 factory.go:153] Registering CRI-O factory Sep 09 22:58:59.728703 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.728587 2578 factory.go:223] Registration of the crio container factory successfully Sep 09 22:58:59.728703 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.728641 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 09 22:58:59.728703 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.728660 2578 factory.go:103] Registering Raw factory Sep 09 22:58:59.728703 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.728676 2578 manager.go:1196] Started watching for new ooms in manager Sep 09 22:58:59.729625 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.729606 2578 manager.go:349] Starting recovery of all containers Sep 09 22:58:59.730522 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.730293 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 22:58:59.732525 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:58:59.732504 2578 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-252.ec2.internal\" not found" node="ip-10-0-139-252.ec2.internal" Sep 09 22:58:59.739591 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.739577 2578 manager.go:354] Recovery completed Sep 09 22:58:59.743550 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.743484 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 22:58:59.745627 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.745612 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-252.ec2.internal" event="NodeHasSufficientMemory" Sep 09 22:58:59.745735 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.745646 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-252.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 22:58:59.745735 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.745661 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-252.ec2.internal" event="NodeHasSufficientPID" Sep 09 22:58:59.746225 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.746214 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 09 22:58:59.746225 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.746221 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 22:58:59.746225 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.746225 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 09 22:58:59.746324 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.746241 2578 state_mem.go:36] "Initialized new in-memory state store" Sep 09 22:58:59.746884 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.746868 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-252.ec2.internal" event="NodeHasSufficientMemory" Sep 09 22:58:59.746949 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.746898 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-252.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 22:58:59.746949 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.746913 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-252.ec2.internal" event="NodeHasSufficientPID" Sep 09 22:58:59.748794 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.748783 2578 policy_none.go:49] "None policy: Start" Sep 09 22:58:59.748837 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.748799 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 09 22:58:59.748837 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.748809 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 09 22:58:59.750714 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.750698 2578 policy_none.go:47] "Start" Sep 09 22:58:59.793964 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.793950 2578 manager.go:337] "Starting Device Plugin manager" Sep 09 22:58:59.812252 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:58:59.793988 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 09 22:58:59.812252 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.793999 2578 server.go:87] "Starting device plugin registration server" Sep 09 22:58:59.812252 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.794342 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 09 22:58:59.812252 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.794351 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 09 22:58:59.812252 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.794448 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 09 22:58:59.812252 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.794542 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 09 22:58:59.812252 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.794549 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 09 22:58:59.812252 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:58:59.795004 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 09 22:58:59.812252 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:58:59.795043 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-252.ec2.internal\" not found" Sep 09 22:58:59.843959 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.843915 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 09 22:58:59.845296 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.845277 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 09 22:58:59.845406 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.845325 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 09 22:58:59.845406 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.845358 2578 kubelet.go:2448] "Starting kubelet main sync loop" Sep 09 22:58:59.845487 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:58:59.845425 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 09 22:58:59.847571 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.847548 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 22:58:59.895416 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.895390 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 22:58:59.896286 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.896268 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-252.ec2.internal" event="NodeHasSufficientMemory" Sep 09 22:58:59.896365 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.896302 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-252.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 22:58:59.896365 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.896313 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-252.ec2.internal" event="NodeHasSufficientPID" Sep 09 22:58:59.896365 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.896334 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-252.ec2.internal" Sep 09 22:58:59.904831 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.904817 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-252.ec2.internal" Sep 09 22:58:59.904883 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:58:59.904840 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-252.ec2.internal\": node \"ip-10-0-139-252.ec2.internal\" not found" Sep 09 22:58:59.918096 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:58:59.918077 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-252.ec2.internal\" not found" Sep 09 22:58:59.946454 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.946408 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-252.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-252.ec2.internal"] Sep 09 22:58:59.946539 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.946498 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 22:58:59.947356 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.947343 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-252.ec2.internal" event="NodeHasSufficientMemory" Sep 09 22:58:59.947455 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.947368 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-252.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 22:58:59.947455 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.947394 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-252.ec2.internal" event="NodeHasSufficientPID" Sep 09 22:58:59.949570 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.949559 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 22:58:59.949690 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.949675 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-252.ec2.internal" Sep 09 22:58:59.949762 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.949721 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 22:58:59.950265 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.950248 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-252.ec2.internal" event="NodeHasSufficientMemory" Sep 09 22:58:59.950354 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.950277 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-252.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 22:58:59.950354 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.950287 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-252.ec2.internal" event="NodeHasSufficientPID" Sep 09 22:58:59.950354 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.950258 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-252.ec2.internal" event="NodeHasSufficientMemory" Sep 09 22:58:59.950354 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.950338 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-252.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 22:58:59.950354 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.950347 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-252.ec2.internal" event="NodeHasSufficientPID" Sep 09 22:58:59.952435 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.952420 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-252.ec2.internal" Sep 09 22:58:59.952526 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.952449 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 22:58:59.953104 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.953091 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-252.ec2.internal" event="NodeHasSufficientMemory" Sep 09 22:58:59.953172 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.953116 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-252.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 22:58:59.953172 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:58:59.953130 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-252.ec2.internal" event="NodeHasSufficientPID" Sep 09 22:58:59.975964 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:58:59.975944 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-252.ec2.internal\" not found" node="ip-10-0-139-252.ec2.internal" Sep 09 22:58:59.980475 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:58:59.980462 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-252.ec2.internal\" not found" node="ip-10-0-139-252.ec2.internal" Sep 09 22:59:00.018984 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:00.018962 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-252.ec2.internal\" not found" Sep 09 22:59:00.029312 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:00.029292 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebaf3d05ceef927b72c983f2c9e143f4-config\") pod \"kube-apiserver-proxy-ip-10-0-139-252.ec2.internal\" (UID: \"ebaf3d05ceef927b72c983f2c9e143f4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-252.ec2.internal" Sep 09 22:59:00.029411 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:00.029317 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3679e0599d5713d056ddee61edfd78b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-252.ec2.internal\" (UID: \"c3679e0599d5713d056ddee61edfd78b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-252.ec2.internal" Sep 09 22:59:00.029411 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:00.029336 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3679e0599d5713d056ddee61edfd78b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-252.ec2.internal\" (UID: \"c3679e0599d5713d056ddee61edfd78b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-252.ec2.internal" Sep 09 22:59:00.119801 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:00.119763 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-252.ec2.internal\" not found" Sep 09 22:59:00.130115 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:00.130087 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebaf3d05ceef927b72c983f2c9e143f4-config\") pod \"kube-apiserver-proxy-ip-10-0-139-252.ec2.internal\" (UID: \"ebaf3d05ceef927b72c983f2c9e143f4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-252.ec2.internal" Sep 09 22:59:00.130194 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:00.130122 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3679e0599d5713d056ddee61edfd78b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-252.ec2.internal\" (UID: \"c3679e0599d5713d056ddee61edfd78b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-252.ec2.internal" Sep 09 22:59:00.130194 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:00.130142 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3679e0599d5713d056ddee61edfd78b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-252.ec2.internal\" (UID: \"c3679e0599d5713d056ddee61edfd78b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-252.ec2.internal" Sep 09 22:59:00.130194 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:00.130159 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebaf3d05ceef927b72c983f2c9e143f4-config\") pod \"kube-apiserver-proxy-ip-10-0-139-252.ec2.internal\" (UID: \"ebaf3d05ceef927b72c983f2c9e143f4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-252.ec2.internal" Sep 09 22:59:00.130292 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:00.130196 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3679e0599d5713d056ddee61edfd78b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-252.ec2.internal\" (UID: \"c3679e0599d5713d056ddee61edfd78b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-252.ec2.internal" Sep 09 22:59:00.130292 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:00.130205 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3679e0599d5713d056ddee61edfd78b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-252.ec2.internal\" (UID: \"c3679e0599d5713d056ddee61edfd78b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-252.ec2.internal" Sep 09 22:59:00.220451 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:00.220421 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-252.ec2.internal\" not found" Sep 09 22:59:00.278980 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:00.278903 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-252.ec2.internal" Sep 09 22:59:00.283923 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:00.283841 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-252.ec2.internal" Sep 09 22:59:00.321426 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:00.321372 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-252.ec2.internal\" not found" Sep 09 22:59:00.421891 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:00.421854 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-252.ec2.internal\" not found" Sep 09 22:59:00.522435 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:00.522403 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-252.ec2.internal\" not found" Sep 09 22:59:00.612483 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:00.612394 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 22:59:00.622957 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:00.622932 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-252.ec2.internal\" not found" Sep 09 22:59:00.642467 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:00.642439 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 09 22:59:00.642584 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:00.642567 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 22:59:00.642633 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:00.642609 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 22:59:00.642666 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:00.642609 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 22:59:00.710938 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:00.710875 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 22:54:00 +0000 UTC" deadline="2028-06-04 00:36:17.32872313 +0000 UTC" Sep 09 22:59:00.710938 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:00.710927 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15193h37m16.617800468s" Sep 09 22:59:00.723104 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:00.723079 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-252.ec2.internal\" not found" Sep 09 22:59:00.823861 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:00.823823 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-252.ec2.internal\" not found" Sep 09 22:59:00.866949 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:59:00.866892 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebaf3d05ceef927b72c983f2c9e143f4.slice/crio-06ac260c2e127d1a2797fb6b5af67c360dbf8870b432744fb3a0c2dd9b4eca2a WatchSource:0}: Error finding container 06ac260c2e127d1a2797fb6b5af67c360dbf8870b432744fb3a0c2dd9b4eca2a: Status 404 returned error can't find the container with id 06ac260c2e127d1a2797fb6b5af67c360dbf8870b432744fb3a0c2dd9b4eca2a Sep 09 22:59:00.872374 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:00.872358 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 22:59:00.878164 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:59:00.878135 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3679e0599d5713d056ddee61edfd78b.slice/crio-775e59b3e5b0448e70bd173125272741d0548b52adfb2fb39becedc5f0333bb3 WatchSource:0}: Error finding container 775e59b3e5b0448e70bd173125272741d0548b52adfb2fb39becedc5f0333bb3: Status 404 returned error can't find the container with id 775e59b3e5b0448e70bd173125272741d0548b52adfb2fb39becedc5f0333bb3 Sep 09 22:59:00.924744 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:00.924695 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-252.ec2.internal\" not found" Sep 09 22:59:01.025273 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:01.025244 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-252.ec2.internal\" not found" Sep 09 22:59:01.125849 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:01.125771 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-252.ec2.internal\" not found" Sep 09 22:59:01.150934 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.150907 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 22:59:01.227742 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.227708 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-252.ec2.internal" Sep 09 22:59:01.239803 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.239772 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 22:59:01.240810 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.240789 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-252.ec2.internal" Sep 09 22:59:01.250218 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.250194 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 22:59:01.700444 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.700403 2578 apiserver.go:52] "Watching apiserver" Sep 09 22:59:01.710406 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.710377 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 09 22:59:01.711594 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.711559 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-4vtjn","kube-system/konnectivity-agent-wc456","kube-system/kube-apiserver-proxy-ip-10-0-139-252.ec2.internal","openshift-cluster-node-tuning-operator/tuned-b8khc","openshift-dns/node-resolver-6j767","openshift-image-registry/node-ca-75rcr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-252.ec2.internal","openshift-multus/multus-additional-cni-plugins-kpcwm","openshift-network-diagnostics/network-check-target-mkgjx","kube-system/global-pull-secret-syncer-b6l2s","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2","openshift-multus/multus-dsd8k","openshift-multus/network-metrics-daemon-l9h9s","openshift-network-operator/iptables-alerter-zbjsz"] Sep 09 22:59:01.714886 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.714860 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:01.715024 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:01.714954 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6l2s" podUID="55304a42-5933-48eb-bbbd-ed2f0d4b933b" Sep 09 22:59:01.717574 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.717549 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:01.717696 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:01.717631 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9h9s" podUID="185f1e13-d2f1-4c57-99d8-c3eb63e20512" Sep 09 22:59:01.720031 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.720006 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zbjsz" Sep 09 22:59:01.722168 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.722144 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.726816 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.726785 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 09 22:59:01.729002 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.727089 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5w6nn\"" Sep 09 22:59:01.729002 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.727471 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 09 22:59:01.729002 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.727765 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 09 22:59:01.729002 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.728020 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-h25jx\"" Sep 09 22:59:01.729002 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.728206 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 09 22:59:01.729002 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.728656 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6j767" Sep 09 22:59:01.729582 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.729558 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 09 22:59:01.729827 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.729813 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 09 22:59:01.730421 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.729848 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-75rcr" Sep 09 22:59:01.730421 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.729863 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:01.730557 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:01.730418 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkgjx" podUID="b52163f6-1997-47c2-a86a-88a1cecaf4b4" Sep 09 22:59:01.732733 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.732716 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kpcwm" Sep 09 22:59:01.734994 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.734971 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dt8l4\"" Sep 09 22:59:01.735109 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.735024 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 09 22:59:01.735830 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.735263 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 09 22:59:01.735830 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.735324 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 09 22:59:01.735830 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.735400 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 09 22:59:01.735830 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.735586 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 09 22:59:01.735830 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.735629 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wc456" Sep 09 22:59:01.735830 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.735589 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7gnqk\"" Sep 09 22:59:01.737517 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.737481 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 09 22:59:01.737627 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.737540 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 09 22:59:01.737943 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.737894 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 09 22:59:01.738040 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.737906 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vpwzc\"" Sep 09 22:59:01.738109 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.738063 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 09 22:59:01.738341 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.738324 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 09 22:59:01.738499 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.738482 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lp6zf\"" Sep 09 22:59:01.738589 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.738571 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 09 22:59:01.739010 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.738877 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 09 22:59:01.740013 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.739994 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.740862 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.740842 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/df484634-6144-4f7a-bae1-3e7826c83e7b-iptables-alerter-script\") pod \"iptables-alerter-zbjsz\" (UID: \"df484634-6144-4f7a-bae1-3e7826c83e7b\") " pod="openshift-network-operator/iptables-alerter-zbjsz" Sep 09 22:59:01.740958 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.740878 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-run\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.740958 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.740925 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f15552a-743f-44be-ae80-73ade41eba6f-tmp\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.740958 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.740950 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs\") pod \"network-metrics-daemon-l9h9s\" (UID: \"185f1e13-d2f1-4c57-99d8-c3eb63e20512\") " pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:01.741121 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.740980 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-etc-sysctl-conf\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.741121 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.741006 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-etc-systemd\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.741121 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.741030 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-lib-modules\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.741121 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.741058 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-host\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.741121 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.741084 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ctgq\" (UniqueName: \"kubernetes.io/projected/7f15552a-743f-44be-ae80-73ade41eba6f-kube-api-access-5ctgq\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.741121 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.741107 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c864525a-b22a-4fef-ba7e-4b9541cbd717-host\") pod \"node-ca-75rcr\" (UID: \"c864525a-b22a-4fef-ba7e-4b9541cbd717\") " pod="openshift-image-registry/node-ca-75rcr" Sep 09 22:59:01.741422 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.741149 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c864525a-b22a-4fef-ba7e-4b9541cbd717-serviceca\") pod \"node-ca-75rcr\" (UID: \"c864525a-b22a-4fef-ba7e-4b9541cbd717\") " pod="openshift-image-registry/node-ca-75rcr" Sep 09 22:59:01.741422 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.741260 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pb2r\" (UniqueName: \"kubernetes.io/projected/b52163f6-1997-47c2-a86a-88a1cecaf4b4-kube-api-access-7pb2r\") pod \"network-check-target-mkgjx\" (UID: \"b52163f6-1997-47c2-a86a-88a1cecaf4b4\") " pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:01.741422 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.741297 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-etc-kubernetes\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.741422 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.741324 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-var-lib-kubelet\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.741422 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.741359 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7f15552a-743f-44be-ae80-73ade41eba6f-etc-tuned\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.741422 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.741420 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz6mq\" (UniqueName: \"kubernetes.io/projected/185f1e13-d2f1-4c57-99d8-c3eb63e20512-kube-api-access-nz6mq\") pod \"network-metrics-daemon-l9h9s\" (UID: \"185f1e13-d2f1-4c57-99d8-c3eb63e20512\") " pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:01.741800 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.741446 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df484634-6144-4f7a-bae1-3e7826c83e7b-host-slash\") pod \"iptables-alerter-zbjsz\" (UID: \"df484634-6144-4f7a-bae1-3e7826c83e7b\") " pod="openshift-network-operator/iptables-alerter-zbjsz" Sep 09 22:59:01.741800 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.741472 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6c9f70f3-174f-4bcf-9bb3-838910f4fce9-hosts-file\") pod \"node-resolver-6j767\" (UID: \"6c9f70f3-174f-4bcf-9bb3-838910f4fce9\") " pod="openshift-dns/node-resolver-6j767" Sep 09 22:59:01.741800 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.741496 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c9f70f3-174f-4bcf-9bb3-838910f4fce9-tmp-dir\") pod \"node-resolver-6j767\" (UID: \"6c9f70f3-174f-4bcf-9bb3-838910f4fce9\") " pod="openshift-dns/node-resolver-6j767" Sep 09 22:59:01.741800 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.741536 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-sys\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.741800 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.741559 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/55304a42-5933-48eb-bbbd-ed2f0d4b933b-dbus\") pod \"global-pull-secret-syncer-b6l2s\" (UID: \"55304a42-5933-48eb-bbbd-ed2f0d4b933b\") " pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:01.741800 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.741592 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55304a42-5933-48eb-bbbd-ed2f0d4b933b-original-pull-secret\") pod \"global-pull-secret-syncer-b6l2s\" (UID: \"55304a42-5933-48eb-bbbd-ed2f0d4b933b\") " pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:01.741800 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.741617 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjpk2\" (UniqueName: \"kubernetes.io/projected/df484634-6144-4f7a-bae1-3e7826c83e7b-kube-api-access-sjpk2\") pod \"iptables-alerter-zbjsz\" (UID: \"df484634-6144-4f7a-bae1-3e7826c83e7b\") " pod="openshift-network-operator/iptables-alerter-zbjsz" Sep 09 22:59:01.741800 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.741657 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-etc-sysctl-d\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.741800 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.741683 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-etc-modprobe-d\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.741800 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.741722 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-etc-sysconfig\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.741800 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.741764 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/55304a42-5933-48eb-bbbd-ed2f0d4b933b-kubelet-config\") pod \"global-pull-secret-syncer-b6l2s\" (UID: \"55304a42-5933-48eb-bbbd-ed2f0d4b933b\") " pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:01.741800 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.741793 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7hr5\" (UniqueName: \"kubernetes.io/projected/6c9f70f3-174f-4bcf-9bb3-838910f4fce9-kube-api-access-s7hr5\") pod \"node-resolver-6j767\" (UID: \"6c9f70f3-174f-4bcf-9bb3-838910f4fce9\") " pod="openshift-dns/node-resolver-6j767" Sep 09 22:59:01.742338 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.741819 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlrvg\" (UniqueName: \"kubernetes.io/projected/c864525a-b22a-4fef-ba7e-4b9541cbd717-kube-api-access-rlrvg\") pod \"node-ca-75rcr\" (UID: \"c864525a-b22a-4fef-ba7e-4b9541cbd717\") " pod="openshift-image-registry/node-ca-75rcr" Sep 09 22:59:01.742644 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.742622 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 09 22:59:01.742864 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.742845 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 09 22:59:01.743036 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.743020 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zrrrc\"" Sep 09 22:59:01.743462 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.743413 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 09 22:59:01.743646 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.743630 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 09 22:59:01.743795 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.743780 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 09 22:59:01.744008 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.743991 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 09 22:59:01.744223 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.744206 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 22:59:01.744414 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.744399 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" Sep 09 22:59:01.745846 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.745743 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.746991 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.746972 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 09 22:59:01.747123 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.747108 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nb5rh\"" Sep 09 22:59:01.747414 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.747202 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 09 22:59:01.747414 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.747207 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 09 22:59:01.748145 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.748122 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-l7h7r\"" Sep 09 22:59:01.748237 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.748144 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 09 22:59:01.790031 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.790003 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-twfdc" Sep 09 22:59:01.797556 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.797524 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-twfdc" Sep 09 22:59:01.829875 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.829837 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 09 22:59:01.842489 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.842443 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlrvg\" (UniqueName: \"kubernetes.io/projected/c864525a-b22a-4fef-ba7e-4b9541cbd717-kube-api-access-rlrvg\") pod \"node-ca-75rcr\" (UID: \"c864525a-b22a-4fef-ba7e-4b9541cbd717\") " pod="openshift-image-registry/node-ca-75rcr" Sep 09 22:59:01.842489 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.842479 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-etc-kubernetes\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.842721 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.842503 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3c974c39-606b-4f5d-ae55-cd15d6393611-agent-certs\") pod \"konnectivity-agent-wc456\" (UID: \"3c974c39-606b-4f5d-ae55-cd15d6393611\") " pod="kube-system/konnectivity-agent-wc456" Sep 09 22:59:01.842721 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.842528 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-host-run-ovn-kubernetes\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.842721 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.842554 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c3cbf093-0861-4893-8d8c-8ca3d30987c2-ovnkube-config\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.842721 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.842587 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-multus-cni-dir\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.842721 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.842625 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-cni-binary-copy\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.842721 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.842656 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-host-run-multus-certs\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.842721 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.842679 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5e113c63-fae1-445e-872d-ca295524db02-registration-dir\") pod \"aws-ebs-csi-driver-node-lhjb2\" (UID: \"5e113c63-fae1-445e-872d-ca295524db02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" Sep 09 22:59:01.842721 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.842702 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqkt6\" (UniqueName: \"kubernetes.io/projected/5e113c63-fae1-445e-872d-ca295524db02-kube-api-access-fqkt6\") pod \"aws-ebs-csi-driver-node-lhjb2\" (UID: \"5e113c63-fae1-445e-872d-ca295524db02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" Sep 09 22:59:01.843041 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.842728 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-host-slash\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.843041 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.842748 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5e113c63-fae1-445e-872d-ca295524db02-etc-selinux\") pod \"aws-ebs-csi-driver-node-lhjb2\" (UID: \"5e113c63-fae1-445e-872d-ca295524db02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" Sep 09 22:59:01.843041 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.842773 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-etc-sysctl-conf\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.843041 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.842843 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-etc-systemd\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.843041 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.842888 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-host\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.843041 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.842930 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-etc-sysctl-conf\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.843041 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.842930 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-host\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.843041 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.842975 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-etc-systemd\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.843041 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.842995 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c864525a-b22a-4fef-ba7e-4b9541cbd717-host\") pod \"node-ca-75rcr\" (UID: \"c864525a-b22a-4fef-ba7e-4b9541cbd717\") " pod="openshift-image-registry/node-ca-75rcr" Sep 09 22:59:01.843041 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843011 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c864525a-b22a-4fef-ba7e-4b9541cbd717-serviceca\") pod \"node-ca-75rcr\" (UID: \"c864525a-b22a-4fef-ba7e-4b9541cbd717\") " pod="openshift-image-registry/node-ca-75rcr" Sep 09 22:59:01.843041 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843027 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pb2r\" (UniqueName: \"kubernetes.io/projected/b52163f6-1997-47c2-a86a-88a1cecaf4b4-kube-api-access-7pb2r\") pod \"network-check-target-mkgjx\" (UID: \"b52163f6-1997-47c2-a86a-88a1cecaf4b4\") " pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:01.843041 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843043 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df484634-6144-4f7a-bae1-3e7826c83e7b-host-slash\") pod \"iptables-alerter-zbjsz\" (UID: \"df484634-6144-4f7a-bae1-3e7826c83e7b\") " pod="openshift-network-operator/iptables-alerter-zbjsz" Sep 09 22:59:01.843646 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843059 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c9f70f3-174f-4bcf-9bb3-838910f4fce9-tmp-dir\") pod \"node-resolver-6j767\" (UID: \"6c9f70f3-174f-4bcf-9bb3-838910f4fce9\") " pod="openshift-dns/node-resolver-6j767" Sep 09 22:59:01.843646 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843081 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-run-openvswitch\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.843646 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843092 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c864525a-b22a-4fef-ba7e-4b9541cbd717-host\") pod \"node-ca-75rcr\" (UID: \"c864525a-b22a-4fef-ba7e-4b9541cbd717\") " pod="openshift-image-registry/node-ca-75rcr" Sep 09 22:59:01.843646 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843103 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c3cbf093-0861-4893-8d8c-8ca3d30987c2-env-overrides\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.843646 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843131 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df484634-6144-4f7a-bae1-3e7826c83e7b-host-slash\") pod \"iptables-alerter-zbjsz\" (UID: \"df484634-6144-4f7a-bae1-3e7826c83e7b\") " pod="openshift-network-operator/iptables-alerter-zbjsz" Sep 09 22:59:01.843646 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843160 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec88f986-9c08-4e93-89c5-919467243b6f-system-cni-dir\") pod \"multus-additional-cni-plugins-kpcwm\" (UID: \"ec88f986-9c08-4e93-89c5-919467243b6f\") " pod="openshift-multus/multus-additional-cni-plugins-kpcwm" Sep 09 22:59:01.843646 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843193 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-host-kubelet\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.843646 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843219 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-systemd-units\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.843646 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843245 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec88f986-9c08-4e93-89c5-919467243b6f-cnibin\") pod \"multus-additional-cni-plugins-kpcwm\" (UID: \"ec88f986-9c08-4e93-89c5-919467243b6f\") " pod="openshift-multus/multus-additional-cni-plugins-kpcwm" Sep 09 22:59:01.843646 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843270 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lspgg\" (UniqueName: \"kubernetes.io/projected/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-kube-api-access-lspgg\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.843646 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843290 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5e113c63-fae1-445e-872d-ca295524db02-device-dir\") pod \"aws-ebs-csi-driver-node-lhjb2\" (UID: \"5e113c63-fae1-445e-872d-ca295524db02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" Sep 09 22:59:01.843646 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843308 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-sys\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.843646 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843357 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-sys\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.843646 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55304a42-5933-48eb-bbbd-ed2f0d4b933b-original-pull-secret\") pod \"global-pull-secret-syncer-b6l2s\" (UID: \"55304a42-5933-48eb-bbbd-ed2f0d4b933b\") " pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:01.843646 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843437 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-run-systemd\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.843646 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843454 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jswnl\" (UniqueName: \"kubernetes.io/projected/c3cbf093-0861-4893-8d8c-8ca3d30987c2-kube-api-access-jswnl\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.843646 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843467 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c9f70f3-174f-4bcf-9bb3-838910f4fce9-tmp-dir\") pod \"node-resolver-6j767\" (UID: \"6c9f70f3-174f-4bcf-9bb3-838910f4fce9\") " pod="openshift-dns/node-resolver-6j767" Sep 09 22:59:01.844533 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843476 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3c974c39-606b-4f5d-ae55-cd15d6393611-konnectivity-ca\") pod \"konnectivity-agent-wc456\" (UID: \"3c974c39-606b-4f5d-ae55-cd15d6393611\") " pod="kube-system/konnectivity-agent-wc456" Sep 09 22:59:01.844533 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:01.843549 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 22:59:01.844533 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843589 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5e113c63-fae1-445e-872d-ca295524db02-sys-fs\") pod \"aws-ebs-csi-driver-node-lhjb2\" (UID: \"5e113c63-fae1-445e-872d-ca295524db02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" Sep 09 22:59:01.844533 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:01.843610 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55304a42-5933-48eb-bbbd-ed2f0d4b933b-original-pull-secret podName:55304a42-5933-48eb-bbbd-ed2f0d4b933b nodeName:}" failed. No retries permitted until 2026-09-09 22:59:02.343588923 +0000 UTC m=+3.094173860 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55304a42-5933-48eb-bbbd-ed2f0d4b933b-original-pull-secret") pod "global-pull-secret-syncer-b6l2s" (UID: "55304a42-5933-48eb-bbbd-ed2f0d4b933b") : object "kube-system"/"original-pull-secret" not registered Sep 09 22:59:01.844533 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843615 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c864525a-b22a-4fef-ba7e-4b9541cbd717-serviceca\") pod \"node-ca-75rcr\" (UID: \"c864525a-b22a-4fef-ba7e-4b9541cbd717\") " pod="openshift-image-registry/node-ca-75rcr" Sep 09 22:59:01.844533 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843657 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-host-cni-netd\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.844533 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843698 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-etc-sysctl-d\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.844533 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843732 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-node-log\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.844533 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843765 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5e113c63-fae1-445e-872d-ca295524db02-socket-dir\") pod \"aws-ebs-csi-driver-node-lhjb2\" (UID: \"5e113c63-fae1-445e-872d-ca295524db02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" Sep 09 22:59:01.844533 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-etc-sysconfig\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.844533 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843820 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-etc-sysctl-d\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.844533 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843820 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7hr5\" (UniqueName: \"kubernetes.io/projected/6c9f70f3-174f-4bcf-9bb3-838910f4fce9-kube-api-access-s7hr5\") pod \"node-resolver-6j767\" (UID: \"6c9f70f3-174f-4bcf-9bb3-838910f4fce9\") " pod="openshift-dns/node-resolver-6j767" Sep 09 22:59:01.844533 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843847 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-host-run-netns\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.844533 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843866 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c3cbf093-0861-4893-8d8c-8ca3d30987c2-ovn-node-metrics-cert\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.844533 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843885 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-etc-sysconfig\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.844533 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843908 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-system-cni-dir\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.844533 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843951 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/df484634-6144-4f7a-bae1-3e7826c83e7b-iptables-alerter-script\") pod \"iptables-alerter-zbjsz\" (UID: \"df484634-6144-4f7a-bae1-3e7826c83e7b\") " pod="openshift-network-operator/iptables-alerter-zbjsz" Sep 09 22:59:01.845251 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.843974 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-hostroot\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.845251 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844001 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmpwf\" (UniqueName: \"kubernetes.io/projected/ec88f986-9c08-4e93-89c5-919467243b6f-kube-api-access-tmpwf\") pod \"multus-additional-cni-plugins-kpcwm\" (UID: \"ec88f986-9c08-4e93-89c5-919467243b6f\") " pod="openshift-multus/multus-additional-cni-plugins-kpcwm" Sep 09 22:59:01.845251 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844028 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-run-ovn\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.845251 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844044 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-run\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.845251 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844064 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f15552a-743f-44be-ae80-73ade41eba6f-tmp\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.845251 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844100 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs\") pod \"network-metrics-daemon-l9h9s\" (UID: \"185f1e13-d2f1-4c57-99d8-c3eb63e20512\") " pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:01.845251 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844127 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-run\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.845251 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844134 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec88f986-9c08-4e93-89c5-919467243b6f-cni-binary-copy\") pod \"multus-additional-cni-plugins-kpcwm\" (UID: \"ec88f986-9c08-4e93-89c5-919467243b6f\") " pod="openshift-multus/multus-additional-cni-plugins-kpcwm" Sep 09 22:59:01.845251 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:01.844217 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 22:59:01.845251 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844233 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec88f986-9c08-4e93-89c5-919467243b6f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kpcwm\" (UID: \"ec88f986-9c08-4e93-89c5-919467243b6f\") " pod="openshift-multus/multus-additional-cni-plugins-kpcwm" Sep 09 22:59:01.845251 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844261 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-host-run-k8s-cni-cncf-io\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.845251 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:01.844274 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs podName:185f1e13-d2f1-4c57-99d8-c3eb63e20512 nodeName:}" failed. No retries permitted until 2026-09-09 22:59:02.344258191 +0000 UTC m=+3.094843104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs") pod "network-metrics-daemon-l9h9s" (UID: "185f1e13-d2f1-4c57-99d8-c3eb63e20512") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 22:59:01.845251 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844300 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-host-var-lib-cni-bin\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.845251 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844330 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.845251 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844357 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-lib-modules\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.845251 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844377 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ctgq\" (UniqueName: \"kubernetes.io/projected/7f15552a-743f-44be-ae80-73ade41eba6f-kube-api-access-5ctgq\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.845795 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844417 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz6mq\" (UniqueName: \"kubernetes.io/projected/185f1e13-d2f1-4c57-99d8-c3eb63e20512-kube-api-access-nz6mq\") pod \"network-metrics-daemon-l9h9s\" (UID: \"185f1e13-d2f1-4c57-99d8-c3eb63e20512\") " pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:01.845795 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844441 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-etc-openvswitch\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.845795 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844482 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-log-socket\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.845795 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844504 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-cnibin\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.845795 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844531 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-var-lib-openvswitch\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.845795 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844553 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-etc-kubernetes\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.845795 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844573 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-var-lib-kubelet\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.845795 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844582 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-lib-modules\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.845795 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844595 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7f15552a-743f-44be-ae80-73ade41eba6f-etc-tuned\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.845795 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844604 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/df484634-6144-4f7a-bae1-3e7826c83e7b-iptables-alerter-script\") pod \"iptables-alerter-zbjsz\" (UID: \"df484634-6144-4f7a-bae1-3e7826c83e7b\") " pod="openshift-network-operator/iptables-alerter-zbjsz" Sep 09 22:59:01.845795 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844616 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6c9f70f3-174f-4bcf-9bb3-838910f4fce9-hosts-file\") pod \"node-resolver-6j767\" (UID: \"6c9f70f3-174f-4bcf-9bb3-838910f4fce9\") " pod="openshift-dns/node-resolver-6j767" Sep 09 22:59:01.845795 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844641 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ec88f986-9c08-4e93-89c5-919467243b6f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kpcwm\" (UID: \"ec88f986-9c08-4e93-89c5-919467243b6f\") " pod="openshift-multus/multus-additional-cni-plugins-kpcwm" Sep 09 22:59:01.845795 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844688 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c3cbf093-0861-4893-8d8c-8ca3d30987c2-ovnkube-script-lib\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.845795 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844696 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-var-lib-kubelet\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.845795 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844705 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6c9f70f3-174f-4bcf-9bb3-838910f4fce9-hosts-file\") pod \"node-resolver-6j767\" (UID: \"6c9f70f3-174f-4bcf-9bb3-838910f4fce9\") " pod="openshift-dns/node-resolver-6j767" Sep 09 22:59:01.845795 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844727 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-etc-kubernetes\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.845795 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844759 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5e113c63-fae1-445e-872d-ca295524db02-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lhjb2\" (UID: \"5e113c63-fae1-445e-872d-ca295524db02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" Sep 09 22:59:01.846538 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/55304a42-5933-48eb-bbbd-ed2f0d4b933b-dbus\") pod \"global-pull-secret-syncer-b6l2s\" (UID: \"55304a42-5933-48eb-bbbd-ed2f0d4b933b\") " pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:01.846538 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844818 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjpk2\" (UniqueName: \"kubernetes.io/projected/df484634-6144-4f7a-bae1-3e7826c83e7b-kube-api-access-sjpk2\") pod \"iptables-alerter-zbjsz\" (UID: \"df484634-6144-4f7a-bae1-3e7826c83e7b\") " pod="openshift-network-operator/iptables-alerter-zbjsz" Sep 09 22:59:01.846538 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844840 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-host-var-lib-cni-multus\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.846538 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844837 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 09 22:59:01.846538 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844862 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-multus-daemon-config\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.846538 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844897 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ec88f986-9c08-4e93-89c5-919467243b6f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kpcwm\" (UID: \"ec88f986-9c08-4e93-89c5-919467243b6f\") " pod="openshift-multus/multus-additional-cni-plugins-kpcwm" Sep 09 22:59:01.846538 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844874 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/55304a42-5933-48eb-bbbd-ed2f0d4b933b-dbus\") pod \"global-pull-secret-syncer-b6l2s\" (UID: \"55304a42-5933-48eb-bbbd-ed2f0d4b933b\") " pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:01.846538 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844921 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-host-cni-bin\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.846538 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844944 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-host-var-lib-kubelet\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.846538 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844968 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-etc-modprobe-d\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.846538 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.844985 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/55304a42-5933-48eb-bbbd-ed2f0d4b933b-kubelet-config\") pod \"global-pull-secret-syncer-b6l2s\" (UID: \"55304a42-5933-48eb-bbbd-ed2f0d4b933b\") " pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:01.846538 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.845008 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-multus-socket-dir-parent\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.846538 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.845031 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-host-run-netns\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.846538 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.845053 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-multus-conf-dir\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.846538 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.845061 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/55304a42-5933-48eb-bbbd-ed2f0d4b933b-kubelet-config\") pod \"global-pull-secret-syncer-b6l2s\" (UID: \"55304a42-5933-48eb-bbbd-ed2f0d4b933b\") " pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:01.846538 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.845098 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7f15552a-743f-44be-ae80-73ade41eba6f-etc-modprobe-d\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.849683 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.849635 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7f15552a-743f-44be-ae80-73ade41eba6f-etc-tuned\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.850042 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.850020 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f15552a-743f-44be-ae80-73ade41eba6f-tmp\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.850267 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:01.850250 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 22:59:01.850360 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:01.850273 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 22:59:01.850360 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:01.850286 2578 projected.go:196] Error preparing data for projected volume kube-api-access-7pb2r for pod openshift-network-diagnostics/network-check-target-mkgjx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 22:59:01.850360 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:01.850353 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b52163f6-1997-47c2-a86a-88a1cecaf4b4-kube-api-access-7pb2r podName:b52163f6-1997-47c2-a86a-88a1cecaf4b4 nodeName:}" failed. No retries permitted until 2026-09-09 22:59:02.350335545 +0000 UTC m=+3.100920454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7pb2r" (UniqueName: "kubernetes.io/projected/b52163f6-1997-47c2-a86a-88a1cecaf4b4-kube-api-access-7pb2r") pod "network-check-target-mkgjx" (UID: "b52163f6-1997-47c2-a86a-88a1cecaf4b4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 22:59:01.852609 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.852585 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 22:59:01.853691 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.853671 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7hr5\" (UniqueName: \"kubernetes.io/projected/6c9f70f3-174f-4bcf-9bb3-838910f4fce9-kube-api-access-s7hr5\") pod \"node-resolver-6j767\" (UID: \"6c9f70f3-174f-4bcf-9bb3-838910f4fce9\") " pod="openshift-dns/node-resolver-6j767" Sep 09 22:59:01.854238 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.853930 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-252.ec2.internal" event={"ID":"c3679e0599d5713d056ddee61edfd78b","Type":"ContainerStarted","Data":"775e59b3e5b0448e70bd173125272741d0548b52adfb2fb39becedc5f0333bb3"} Sep 09 22:59:01.854551 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.854498 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlrvg\" (UniqueName: \"kubernetes.io/projected/c864525a-b22a-4fef-ba7e-4b9541cbd717-kube-api-access-rlrvg\") pod \"node-ca-75rcr\" (UID: \"c864525a-b22a-4fef-ba7e-4b9541cbd717\") " pod="openshift-image-registry/node-ca-75rcr" Sep 09 22:59:01.855023 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.854999 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz6mq\" (UniqueName: \"kubernetes.io/projected/185f1e13-d2f1-4c57-99d8-c3eb63e20512-kube-api-access-nz6mq\") pod \"network-metrics-daemon-l9h9s\" (UID: \"185f1e13-d2f1-4c57-99d8-c3eb63e20512\") " pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:01.855468 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.855444 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ctgq\" (UniqueName: \"kubernetes.io/projected/7f15552a-743f-44be-ae80-73ade41eba6f-kube-api-access-5ctgq\") pod \"tuned-b8khc\" (UID: \"7f15552a-743f-44be-ae80-73ade41eba6f\") " pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:01.855556 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.855426 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjpk2\" (UniqueName: \"kubernetes.io/projected/df484634-6144-4f7a-bae1-3e7826c83e7b-kube-api-access-sjpk2\") pod \"iptables-alerter-zbjsz\" (UID: \"df484634-6144-4f7a-bae1-3e7826c83e7b\") " pod="openshift-network-operator/iptables-alerter-zbjsz" Sep 09 22:59:01.855847 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.855827 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-252.ec2.internal" event={"ID":"ebaf3d05ceef927b72c983f2c9e143f4","Type":"ContainerStarted","Data":"06ac260c2e127d1a2797fb6b5af67c360dbf8870b432744fb3a0c2dd9b4eca2a"} Sep 09 22:59:01.946005 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.945971 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-node-log\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.946193 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946008 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5e113c63-fae1-445e-872d-ca295524db02-socket-dir\") pod \"aws-ebs-csi-driver-node-lhjb2\" (UID: \"5e113c63-fae1-445e-872d-ca295524db02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" Sep 09 22:59:01.946193 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946038 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-host-run-netns\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.946193 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946088 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-host-run-netns\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.946193 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946099 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-node-log\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.946193 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946115 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c3cbf093-0861-4893-8d8c-8ca3d30987c2-ovn-node-metrics-cert\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.946193 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946132 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-system-cni-dir\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.946193 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946156 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-hostroot\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.946193 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946174 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmpwf\" (UniqueName: \"kubernetes.io/projected/ec88f986-9c08-4e93-89c5-919467243b6f-kube-api-access-tmpwf\") pod \"multus-additional-cni-plugins-kpcwm\" (UID: \"ec88f986-9c08-4e93-89c5-919467243b6f\") " pod="openshift-multus/multus-additional-cni-plugins-kpcwm" Sep 09 22:59:01.946193 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946194 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5e113c63-fae1-445e-872d-ca295524db02-socket-dir\") pod \"aws-ebs-csi-driver-node-lhjb2\" (UID: \"5e113c63-fae1-445e-872d-ca295524db02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" Sep 09 22:59:01.946193 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-run-ovn\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.946762 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946236 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec88f986-9c08-4e93-89c5-919467243b6f-cni-binary-copy\") pod \"multus-additional-cni-plugins-kpcwm\" (UID: \"ec88f986-9c08-4e93-89c5-919467243b6f\") " pod="openshift-multus/multus-additional-cni-plugins-kpcwm" Sep 09 22:59:01.946762 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946259 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-system-cni-dir\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.946762 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946263 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec88f986-9c08-4e93-89c5-919467243b6f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kpcwm\" (UID: \"ec88f986-9c08-4e93-89c5-919467243b6f\") " pod="openshift-multus/multus-additional-cni-plugins-kpcwm" Sep 09 22:59:01.946762 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946273 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-hostroot\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.946762 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946292 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-run-ovn\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.946762 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946291 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-host-run-k8s-cni-cncf-io\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.946762 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946334 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-host-run-k8s-cni-cncf-io\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.946762 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946410 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-host-var-lib-cni-bin\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.946762 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946465 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-host-var-lib-cni-bin\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.946762 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946503 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.946762 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946526 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec88f986-9c08-4e93-89c5-919467243b6f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kpcwm\" (UID: \"ec88f986-9c08-4e93-89c5-919467243b6f\") " pod="openshift-multus/multus-additional-cni-plugins-kpcwm" Sep 09 22:59:01.946762 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946533 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-etc-openvswitch\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.946762 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946559 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-log-socket\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.946762 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946585 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-cnibin\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.946762 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946594 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.946762 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946561 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-etc-openvswitch\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.946762 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946612 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-log-socket\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.947706 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946614 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-var-lib-openvswitch\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.947706 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946653 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-var-lib-openvswitch\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.947706 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946658 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-cnibin\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.947706 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946657 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ec88f986-9c08-4e93-89c5-919467243b6f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kpcwm\" (UID: \"ec88f986-9c08-4e93-89c5-919467243b6f\") " pod="openshift-multus/multus-additional-cni-plugins-kpcwm" Sep 09 22:59:01.947706 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946697 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c3cbf093-0861-4893-8d8c-8ca3d30987c2-ovnkube-script-lib\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.947706 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946722 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5e113c63-fae1-445e-872d-ca295524db02-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lhjb2\" (UID: \"5e113c63-fae1-445e-872d-ca295524db02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" Sep 09 22:59:01.947706 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946748 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-host-var-lib-cni-multus\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.947706 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-multus-daemon-config\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.947706 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946818 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ec88f986-9c08-4e93-89c5-919467243b6f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kpcwm\" (UID: \"ec88f986-9c08-4e93-89c5-919467243b6f\") " pod="openshift-multus/multus-additional-cni-plugins-kpcwm" Sep 09 22:59:01.947706 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946845 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-host-cni-bin\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.947706 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-host-var-lib-kubelet\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.947706 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946907 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-multus-socket-dir-parent\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.947706 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946931 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-host-run-netns\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.947706 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946953 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-multus-conf-dir\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.947706 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946969 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec88f986-9c08-4e93-89c5-919467243b6f-cni-binary-copy\") pod \"multus-additional-cni-plugins-kpcwm\" (UID: \"ec88f986-9c08-4e93-89c5-919467243b6f\") " pod="openshift-multus/multus-additional-cni-plugins-kpcwm" Sep 09 22:59:01.947706 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.946980 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-etc-kubernetes\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.947706 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3c974c39-606b-4f5d-ae55-cd15d6393611-agent-certs\") pod \"konnectivity-agent-wc456\" (UID: \"3c974c39-606b-4f5d-ae55-cd15d6393611\") " pod="kube-system/konnectivity-agent-wc456" Sep 09 22:59:01.948445 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947033 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-host-run-ovn-kubernetes\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.948445 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947040 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-host-cni-bin\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.948445 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947060 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c3cbf093-0861-4893-8d8c-8ca3d30987c2-ovnkube-config\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.948445 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947085 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5e113c63-fae1-445e-872d-ca295524db02-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lhjb2\" (UID: \"5e113c63-fae1-445e-872d-ca295524db02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" Sep 09 22:59:01.948445 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947086 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-multus-cni-dir\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.948445 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947142 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-host-var-lib-cni-multus\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.948445 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947174 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-host-run-netns\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.948445 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947198 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-multus-cni-dir\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.948445 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947218 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-host-var-lib-kubelet\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.948445 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947227 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c3cbf093-0861-4893-8d8c-8ca3d30987c2-ovnkube-script-lib\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.948445 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947145 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ec88f986-9c08-4e93-89c5-919467243b6f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kpcwm\" (UID: \"ec88f986-9c08-4e93-89c5-919467243b6f\") " pod="openshift-multus/multus-additional-cni-plugins-kpcwm" Sep 09 22:59:01.948445 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947277 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-multus-conf-dir\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.948445 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947280 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-host-run-ovn-kubernetes\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.948445 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947231 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-multus-socket-dir-parent\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.948445 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947145 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-cni-binary-copy\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.948445 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947344 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-host-run-multus-certs\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.948445 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947373 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5e113c63-fae1-445e-872d-ca295524db02-registration-dir\") pod \"aws-ebs-csi-driver-node-lhjb2\" (UID: \"5e113c63-fae1-445e-872d-ca295524db02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" Sep 09 22:59:01.949139 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947402 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqkt6\" (UniqueName: \"kubernetes.io/projected/5e113c63-fae1-445e-872d-ca295524db02-kube-api-access-fqkt6\") pod \"aws-ebs-csi-driver-node-lhjb2\" (UID: \"5e113c63-fae1-445e-872d-ca295524db02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" Sep 09 22:59:01.949139 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947428 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-host-slash\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.949139 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947453 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5e113c63-fae1-445e-872d-ca295524db02-etc-selinux\") pod \"aws-ebs-csi-driver-node-lhjb2\" (UID: \"5e113c63-fae1-445e-872d-ca295524db02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" Sep 09 22:59:01.949139 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947524 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-run-openvswitch\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.949139 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947231 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-etc-kubernetes\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.949139 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947550 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c3cbf093-0861-4893-8d8c-8ca3d30987c2-env-overrides\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.949139 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947622 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec88f986-9c08-4e93-89c5-919467243b6f-system-cni-dir\") pod \"multus-additional-cni-plugins-kpcwm\" (UID: \"ec88f986-9c08-4e93-89c5-919467243b6f\") " pod="openshift-multus/multus-additional-cni-plugins-kpcwm" Sep 09 22:59:01.949139 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947670 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-host-kubelet\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.949139 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947674 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c3cbf093-0861-4893-8d8c-8ca3d30987c2-ovnkube-config\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.949139 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947697 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-systemd-units\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.949139 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947723 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec88f986-9c08-4e93-89c5-919467243b6f-cnibin\") pod \"multus-additional-cni-plugins-kpcwm\" (UID: \"ec88f986-9c08-4e93-89c5-919467243b6f\") " pod="openshift-multus/multus-additional-cni-plugins-kpcwm" Sep 09 22:59:01.949139 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947770 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lspgg\" (UniqueName: \"kubernetes.io/projected/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-kube-api-access-lspgg\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.949139 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947780 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-cni-binary-copy\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.949139 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947799 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5e113c63-fae1-445e-872d-ca295524db02-device-dir\") pod \"aws-ebs-csi-driver-node-lhjb2\" (UID: \"5e113c63-fae1-445e-872d-ca295524db02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" Sep 09 22:59:01.949139 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947804 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec88f986-9c08-4e93-89c5-919467243b6f-cnibin\") pod \"multus-additional-cni-plugins-kpcwm\" (UID: \"ec88f986-9c08-4e93-89c5-919467243b6f\") " pod="openshift-multus/multus-additional-cni-plugins-kpcwm" Sep 09 22:59:01.949139 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947825 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-systemd-units\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.949139 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947783 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-multus-daemon-config\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.949793 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947772 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-host-kubelet\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.949793 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947864 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-host-run-multus-certs\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.949793 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947869 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-run-systemd\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.949793 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947902 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jswnl\" (UniqueName: \"kubernetes.io/projected/c3cbf093-0861-4893-8d8c-8ca3d30987c2-kube-api-access-jswnl\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.949793 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947922 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5e113c63-fae1-445e-872d-ca295524db02-device-dir\") pod \"aws-ebs-csi-driver-node-lhjb2\" (UID: \"5e113c63-fae1-445e-872d-ca295524db02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" Sep 09 22:59:01.949793 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947929 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3c974c39-606b-4f5d-ae55-cd15d6393611-konnectivity-ca\") pod \"konnectivity-agent-wc456\" (UID: \"3c974c39-606b-4f5d-ae55-cd15d6393611\") " pod="kube-system/konnectivity-agent-wc456" Sep 09 22:59:01.949793 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947905 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-run-systemd\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.949793 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947949 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5e113c63-fae1-445e-872d-ca295524db02-registration-dir\") pod \"aws-ebs-csi-driver-node-lhjb2\" (UID: \"5e113c63-fae1-445e-872d-ca295524db02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" Sep 09 22:59:01.949793 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947973 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5e113c63-fae1-445e-872d-ca295524db02-sys-fs\") pod \"aws-ebs-csi-driver-node-lhjb2\" (UID: \"5e113c63-fae1-445e-872d-ca295524db02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" Sep 09 22:59:01.949793 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947976 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-host-slash\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.949793 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947976 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c3cbf093-0861-4893-8d8c-8ca3d30987c2-env-overrides\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.949793 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.947736 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec88f986-9c08-4e93-89c5-919467243b6f-system-cni-dir\") pod \"multus-additional-cni-plugins-kpcwm\" (UID: \"ec88f986-9c08-4e93-89c5-919467243b6f\") " pod="openshift-multus/multus-additional-cni-plugins-kpcwm" Sep 09 22:59:01.949793 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.948009 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-host-cni-netd\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.949793 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.948058 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-run-openvswitch\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.949793 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.948212 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5e113c63-fae1-445e-872d-ca295524db02-etc-selinux\") pod \"aws-ebs-csi-driver-node-lhjb2\" (UID: \"5e113c63-fae1-445e-872d-ca295524db02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" Sep 09 22:59:01.949793 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.948212 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5e113c63-fae1-445e-872d-ca295524db02-sys-fs\") pod \"aws-ebs-csi-driver-node-lhjb2\" (UID: \"5e113c63-fae1-445e-872d-ca295524db02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" Sep 09 22:59:01.949793 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.948335 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c3cbf093-0861-4893-8d8c-8ca3d30987c2-host-cni-netd\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.950322 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.948473 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3c974c39-606b-4f5d-ae55-cd15d6393611-konnectivity-ca\") pod \"konnectivity-agent-wc456\" (UID: \"3c974c39-606b-4f5d-ae55-cd15d6393611\") " pod="kube-system/konnectivity-agent-wc456" Sep 09 22:59:01.950322 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.948566 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ec88f986-9c08-4e93-89c5-919467243b6f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kpcwm\" (UID: \"ec88f986-9c08-4e93-89c5-919467243b6f\") " pod="openshift-multus/multus-additional-cni-plugins-kpcwm" Sep 09 22:59:01.950322 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.949474 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c3cbf093-0861-4893-8d8c-8ca3d30987c2-ovn-node-metrics-cert\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:01.950322 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.949953 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3c974c39-606b-4f5d-ae55-cd15d6393611-agent-certs\") pod \"konnectivity-agent-wc456\" (UID: \"3c974c39-606b-4f5d-ae55-cd15d6393611\") " pod="kube-system/konnectivity-agent-wc456" Sep 09 22:59:01.955830 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.955772 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmpwf\" (UniqueName: \"kubernetes.io/projected/ec88f986-9c08-4e93-89c5-919467243b6f-kube-api-access-tmpwf\") pod \"multus-additional-cni-plugins-kpcwm\" (UID: \"ec88f986-9c08-4e93-89c5-919467243b6f\") " pod="openshift-multus/multus-additional-cni-plugins-kpcwm" Sep 09 22:59:01.956176 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.956153 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lspgg\" (UniqueName: \"kubernetes.io/projected/b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d-kube-api-access-lspgg\") pod \"multus-dsd8k\" (UID: \"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d\") " pod="openshift-multus/multus-dsd8k" Sep 09 22:59:01.957714 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.957693 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqkt6\" (UniqueName: \"kubernetes.io/projected/5e113c63-fae1-445e-872d-ca295524db02-kube-api-access-fqkt6\") pod \"aws-ebs-csi-driver-node-lhjb2\" (UID: \"5e113c63-fae1-445e-872d-ca295524db02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" Sep 09 22:59:01.957800 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:01.957717 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jswnl\" (UniqueName: \"kubernetes.io/projected/c3cbf093-0861-4893-8d8c-8ca3d30987c2-kube-api-access-jswnl\") pod \"ovnkube-node-4vtjn\" (UID: \"c3cbf093-0861-4893-8d8c-8ca3d30987c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:02.039841 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:02.039759 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zbjsz" Sep 09 22:59:02.049444 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:02.049412 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b8khc" Sep 09 22:59:02.059357 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:02.059335 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6j767" Sep 09 22:59:02.064105 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:02.064085 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-75rcr" Sep 09 22:59:02.074696 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:02.074667 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kpcwm" Sep 09 22:59:02.082471 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:02.082445 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wc456" Sep 09 22:59:02.095321 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:02.095299 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:02.099389 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:02.099368 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" Sep 09 22:59:02.106475 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:02.106453 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dsd8k" Sep 09 22:59:02.144399 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:02.144362 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 22:59:02.349564 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:02.349533 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 22:59:02.350378 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:02.350351 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55304a42-5933-48eb-bbbd-ed2f0d4b933b-original-pull-secret\") pod \"global-pull-secret-syncer-b6l2s\" (UID: \"55304a42-5933-48eb-bbbd-ed2f0d4b933b\") " pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:02.350541 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:02.350418 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs\") pod \"network-metrics-daemon-l9h9s\" (UID: \"185f1e13-d2f1-4c57-99d8-c3eb63e20512\") " pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:02.350541 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:02.350518 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 22:59:02.350541 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:02.350530 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 22:59:02.350712 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:02.350595 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs podName:185f1e13-d2f1-4c57-99d8-c3eb63e20512 nodeName:}" failed. No retries permitted until 2026-09-09 22:59:03.350575703 +0000 UTC m=+4.101160609 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs") pod "network-metrics-daemon-l9h9s" (UID: "185f1e13-d2f1-4c57-99d8-c3eb63e20512") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 22:59:02.350712 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:02.350611 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55304a42-5933-48eb-bbbd-ed2f0d4b933b-original-pull-secret podName:55304a42-5933-48eb-bbbd-ed2f0d4b933b nodeName:}" failed. No retries permitted until 2026-09-09 22:59:03.350603824 +0000 UTC m=+4.101188730 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55304a42-5933-48eb-bbbd-ed2f0d4b933b-original-pull-secret") pod "global-pull-secret-syncer-b6l2s" (UID: "55304a42-5933-48eb-bbbd-ed2f0d4b933b") : object "kube-system"/"original-pull-secret" not registered Sep 09 22:59:02.451305 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:02.451275 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pb2r\" (UniqueName: \"kubernetes.io/projected/b52163f6-1997-47c2-a86a-88a1cecaf4b4-kube-api-access-7pb2r\") pod \"network-check-target-mkgjx\" (UID: \"b52163f6-1997-47c2-a86a-88a1cecaf4b4\") " pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:02.451527 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:02.451505 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 22:59:02.451580 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:02.451538 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 22:59:02.451580 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:02.451555 2578 projected.go:196] Error preparing data for projected volume kube-api-access-7pb2r for pod openshift-network-diagnostics/network-check-target-mkgjx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 22:59:02.451648 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:02.451630 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b52163f6-1997-47c2-a86a-88a1cecaf4b4-kube-api-access-7pb2r podName:b52163f6-1997-47c2-a86a-88a1cecaf4b4 nodeName:}" failed. No retries permitted until 2026-09-09 22:59:03.451607562 +0000 UTC m=+4.202192484 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pb2r" (UniqueName: "kubernetes.io/projected/b52163f6-1997-47c2-a86a-88a1cecaf4b4-kube-api-access-7pb2r") pod "network-check-target-mkgjx" (UID: "b52163f6-1997-47c2-a86a-88a1cecaf4b4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 22:59:02.796616 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:59:02.796420 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf484634_6144_4f7a_bae1_3e7826c83e7b.slice/crio-ae8a9e421ac6e2aa227159eb01803be3fd9669ed7e6740335103b8105c00a6d6 WatchSource:0}: Error finding container ae8a9e421ac6e2aa227159eb01803be3fd9669ed7e6740335103b8105c00a6d6: Status 404 returned error can't find the container with id ae8a9e421ac6e2aa227159eb01803be3fd9669ed7e6740335103b8105c00a6d6 Sep 09 22:59:02.801233 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:02.801185 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 22:54:02 +0000 UTC" deadline="2028-05-02 18:54:21.456030256 +0000 UTC" Sep 09 22:59:02.801329 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:02.801234 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14419h55m18.654800431s" Sep 09 22:59:02.801828 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:59:02.801798 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f15552a_743f_44be_ae80_73ade41eba6f.slice/crio-a4f18f0009922f18a3c9379938ff713a215c4351acbe2550ae5e19141c66e987 WatchSource:0}: Error finding container a4f18f0009922f18a3c9379938ff713a215c4351acbe2550ae5e19141c66e987: Status 404 returned error can't find the container with id a4f18f0009922f18a3c9379938ff713a215c4351acbe2550ae5e19141c66e987 Sep 09 22:59:02.829919 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:59:02.829891 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33b6c26_be92_4d99_8cbc_cc2ca4b7f65d.slice/crio-31e8de6bf02d81930f0e448f948103867b236b137a2dc4007c0ecc2e327f83e5 WatchSource:0}: Error finding container 31e8de6bf02d81930f0e448f948103867b236b137a2dc4007c0ecc2e327f83e5: Status 404 returned error can't find the container with id 31e8de6bf02d81930f0e448f948103867b236b137a2dc4007c0ecc2e327f83e5 Sep 09 22:59:02.831297 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:59:02.831259 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e113c63_fae1_445e_872d_ca295524db02.slice/crio-5dc7c933ffcb03e329dafdcb9598ea495febf00996889ad574712dca3aec1964 WatchSource:0}: Error finding container 5dc7c933ffcb03e329dafdcb9598ea495febf00996889ad574712dca3aec1964: Status 404 returned error can't find the container with id 5dc7c933ffcb03e329dafdcb9598ea495febf00996889ad574712dca3aec1964 Sep 09 22:59:02.831954 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:59:02.831927 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc864525a_b22a_4fef_ba7e_4b9541cbd717.slice/crio-e18ee59857cd1c6ee3dab39d01f87752867983bae3b99031ca01caf87222f95f WatchSource:0}: Error finding container e18ee59857cd1c6ee3dab39d01f87752867983bae3b99031ca01caf87222f95f: Status 404 returned error can't find the container with id e18ee59857cd1c6ee3dab39d01f87752867983bae3b99031ca01caf87222f95f Sep 09 22:59:02.832895 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:59:02.832863 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c974c39_606b_4f5d_ae55_cd15d6393611.slice/crio-010d64a66af19d8e2fc73f8e6f8e4fa2afcb66657633810ef54a713d1023b614 WatchSource:0}: Error finding container 010d64a66af19d8e2fc73f8e6f8e4fa2afcb66657633810ef54a713d1023b614: Status 404 returned error can't find the container with id 010d64a66af19d8e2fc73f8e6f8e4fa2afcb66657633810ef54a713d1023b614 Sep 09 22:59:02.834095 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:59:02.834075 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec88f986_9c08_4e93_89c5_919467243b6f.slice/crio-a4c563e492cfed6800c1782761a97030b016c681f0fac9e07bebdcef7689634d WatchSource:0}: Error finding container a4c563e492cfed6800c1782761a97030b016c681f0fac9e07bebdcef7689634d: Status 404 returned error can't find the container with id a4c563e492cfed6800c1782761a97030b016c681f0fac9e07bebdcef7689634d Sep 09 22:59:02.858423 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:02.858379 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b8khc" event={"ID":"7f15552a-743f-44be-ae80-73ade41eba6f","Type":"ContainerStarted","Data":"a4f18f0009922f18a3c9379938ff713a215c4351acbe2550ae5e19141c66e987"} Sep 09 22:59:02.859738 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:02.859695 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zbjsz" event={"ID":"df484634-6144-4f7a-bae1-3e7826c83e7b","Type":"ContainerStarted","Data":"ae8a9e421ac6e2aa227159eb01803be3fd9669ed7e6740335103b8105c00a6d6"} Sep 09 22:59:02.860916 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:02.860895 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kpcwm" event={"ID":"ec88f986-9c08-4e93-89c5-919467243b6f","Type":"ContainerStarted","Data":"a4c563e492cfed6800c1782761a97030b016c681f0fac9e07bebdcef7689634d"} Sep 09 22:59:02.862145 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:02.862123 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6j767" event={"ID":"6c9f70f3-174f-4bcf-9bb3-838910f4fce9","Type":"ContainerStarted","Data":"18eb79b72ee176fc5393967004dfc21d9b9d2c67e88313bf78c0ec9987c1c34c"} Sep 09 22:59:02.866567 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:02.866544 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wc456" event={"ID":"3c974c39-606b-4f5d-ae55-cd15d6393611","Type":"ContainerStarted","Data":"010d64a66af19d8e2fc73f8e6f8e4fa2afcb66657633810ef54a713d1023b614"} Sep 09 22:59:02.871851 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:02.871826 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-75rcr" event={"ID":"c864525a-b22a-4fef-ba7e-4b9541cbd717","Type":"ContainerStarted","Data":"e18ee59857cd1c6ee3dab39d01f87752867983bae3b99031ca01caf87222f95f"} Sep 09 22:59:02.873456 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:02.873433 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" event={"ID":"5e113c63-fae1-445e-872d-ca295524db02","Type":"ContainerStarted","Data":"5dc7c933ffcb03e329dafdcb9598ea495febf00996889ad574712dca3aec1964"} Sep 09 22:59:02.874638 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:02.874608 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dsd8k" event={"ID":"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d","Type":"ContainerStarted","Data":"31e8de6bf02d81930f0e448f948103867b236b137a2dc4007c0ecc2e327f83e5"} Sep 09 22:59:02.875894 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:02.875872 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" event={"ID":"c3cbf093-0861-4893-8d8c-8ca3d30987c2","Type":"ContainerStarted","Data":"23305c6085ec19afcf8c201490b8618c14d13513c36b81276f75a000b146efb1"} Sep 09 22:59:03.359842 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:03.359024 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs\") pod \"network-metrics-daemon-l9h9s\" (UID: \"185f1e13-d2f1-4c57-99d8-c3eb63e20512\") " pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:03.359842 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:03.359123 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55304a42-5933-48eb-bbbd-ed2f0d4b933b-original-pull-secret\") pod \"global-pull-secret-syncer-b6l2s\" (UID: \"55304a42-5933-48eb-bbbd-ed2f0d4b933b\") " pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:03.359842 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:03.359258 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 22:59:03.359842 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:03.359324 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55304a42-5933-48eb-bbbd-ed2f0d4b933b-original-pull-secret podName:55304a42-5933-48eb-bbbd-ed2f0d4b933b nodeName:}" failed. No retries permitted until 2026-09-09 22:59:05.359304356 +0000 UTC m=+6.109889281 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55304a42-5933-48eb-bbbd-ed2f0d4b933b-original-pull-secret") pod "global-pull-secret-syncer-b6l2s" (UID: "55304a42-5933-48eb-bbbd-ed2f0d4b933b") : object "kube-system"/"original-pull-secret" not registered Sep 09 22:59:03.359842 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:03.359414 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 22:59:03.359842 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:03.359472 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs podName:185f1e13-d2f1-4c57-99d8-c3eb63e20512 nodeName:}" failed. No retries permitted until 2026-09-09 22:59:05.359457038 +0000 UTC m=+6.110041962 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs") pod "network-metrics-daemon-l9h9s" (UID: "185f1e13-d2f1-4c57-99d8-c3eb63e20512") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 22:59:03.460922 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:03.459777 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pb2r\" (UniqueName: \"kubernetes.io/projected/b52163f6-1997-47c2-a86a-88a1cecaf4b4-kube-api-access-7pb2r\") pod \"network-check-target-mkgjx\" (UID: \"b52163f6-1997-47c2-a86a-88a1cecaf4b4\") " pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:03.460922 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:03.460095 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 22:59:03.460922 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:03.460122 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 22:59:03.460922 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:03.460142 2578 projected.go:196] Error preparing data for projected volume kube-api-access-7pb2r for pod openshift-network-diagnostics/network-check-target-mkgjx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 22:59:03.460922 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:03.460202 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b52163f6-1997-47c2-a86a-88a1cecaf4b4-kube-api-access-7pb2r podName:b52163f6-1997-47c2-a86a-88a1cecaf4b4 nodeName:}" failed. No retries permitted until 2026-09-09 22:59:05.460183726 +0000 UTC m=+6.210768640 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pb2r" (UniqueName: "kubernetes.io/projected/b52163f6-1997-47c2-a86a-88a1cecaf4b4-kube-api-access-7pb2r") pod "network-check-target-mkgjx" (UID: "b52163f6-1997-47c2-a86a-88a1cecaf4b4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 22:59:03.801796 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:03.801744 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 22:54:02 +0000 UTC" deadline="2028-05-20 20:10:57.489425902 +0000 UTC" Sep 09 22:59:03.801796 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:03.801794 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14853h11m53.68763689s" Sep 09 22:59:03.846651 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:03.846613 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:03.846651 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:03.846650 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:03.846892 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:03.846615 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:03.846892 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:03.846781 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9h9s" podUID="185f1e13-d2f1-4c57-99d8-c3eb63e20512" Sep 09 22:59:03.846892 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:03.846854 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkgjx" podUID="b52163f6-1997-47c2-a86a-88a1cecaf4b4" Sep 09 22:59:03.847232 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:03.846943 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6l2s" podUID="55304a42-5933-48eb-bbbd-ed2f0d4b933b" Sep 09 22:59:03.889412 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:03.889352 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-252.ec2.internal" event={"ID":"ebaf3d05ceef927b72c983f2c9e143f4","Type":"ContainerStarted","Data":"a1ee99ae6c2e8222e8816f82a01b56aff572b6933857ed5efb4ec54c1d2a46b8"} Sep 09 22:59:03.906734 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:03.906684 2578 generic.go:356] "Generic (PLEG): container finished" podID="c3679e0599d5713d056ddee61edfd78b" containerID="c197abe124ae1c2c7eed350cd1db9dcf649270af215e5fa2065c4a8f9e65b8f5" exitCode=0 Sep 09 22:59:03.906973 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:03.906744 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-252.ec2.internal" event={"ID":"c3679e0599d5713d056ddee61edfd78b","Type":"ContainerDied","Data":"c197abe124ae1c2c7eed350cd1db9dcf649270af215e5fa2065c4a8f9e65b8f5"} Sep 09 22:59:03.932400 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:03.932333 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-252.ec2.internal" podStartSLOduration=2.932311014 podStartE2EDuration="2.932311014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 22:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:59:03.904232596 +0000 UTC m=+4.654817529" watchObservedRunningTime="2026-09-09 22:59:03.932311014 +0000 UTC m=+4.682895945" Sep 09 22:59:04.933626 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:04.933554 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-252.ec2.internal" event={"ID":"c3679e0599d5713d056ddee61edfd78b","Type":"ContainerStarted","Data":"1eef8db4d9379b88acc9f5a4116801b36cbb868dc66d8ad15909e6c8b4f0b31d"} Sep 09 22:59:04.950004 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:04.948997 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-252.ec2.internal" podStartSLOduration=3.9489749659999998 podStartE2EDuration="3.948974966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 22:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:59:04.947660453 +0000 UTC m=+5.698245410" watchObservedRunningTime="2026-09-09 22:59:04.948974966 +0000 UTC m=+5.699559899" Sep 09 22:59:05.380759 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:05.380719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55304a42-5933-48eb-bbbd-ed2f0d4b933b-original-pull-secret\") pod \"global-pull-secret-syncer-b6l2s\" (UID: \"55304a42-5933-48eb-bbbd-ed2f0d4b933b\") " pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:05.380965 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:05.380782 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs\") pod \"network-metrics-daemon-l9h9s\" (UID: \"185f1e13-d2f1-4c57-99d8-c3eb63e20512\") " pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:05.380965 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:05.380912 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 22:59:05.381101 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:05.380975 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs podName:185f1e13-d2f1-4c57-99d8-c3eb63e20512 nodeName:}" failed. No retries permitted until 2026-09-09 22:59:09.38095709 +0000 UTC m=+10.131542011 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs") pod "network-metrics-daemon-l9h9s" (UID: "185f1e13-d2f1-4c57-99d8-c3eb63e20512") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 22:59:05.381427 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:05.381406 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 22:59:05.381637 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:05.381620 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55304a42-5933-48eb-bbbd-ed2f0d4b933b-original-pull-secret podName:55304a42-5933-48eb-bbbd-ed2f0d4b933b nodeName:}" failed. No retries permitted until 2026-09-09 22:59:09.381449864 +0000 UTC m=+10.132034770 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55304a42-5933-48eb-bbbd-ed2f0d4b933b-original-pull-secret") pod "global-pull-secret-syncer-b6l2s" (UID: "55304a42-5933-48eb-bbbd-ed2f0d4b933b") : object "kube-system"/"original-pull-secret" not registered Sep 09 22:59:05.481408 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:05.481362 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pb2r\" (UniqueName: \"kubernetes.io/projected/b52163f6-1997-47c2-a86a-88a1cecaf4b4-kube-api-access-7pb2r\") pod \"network-check-target-mkgjx\" (UID: \"b52163f6-1997-47c2-a86a-88a1cecaf4b4\") " pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:05.481685 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:05.481580 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 22:59:05.481685 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:05.481602 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 22:59:05.481685 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:05.481614 2578 projected.go:196] Error preparing data for projected volume kube-api-access-7pb2r for pod openshift-network-diagnostics/network-check-target-mkgjx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 22:59:05.481888 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:05.481704 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b52163f6-1997-47c2-a86a-88a1cecaf4b4-kube-api-access-7pb2r podName:b52163f6-1997-47c2-a86a-88a1cecaf4b4 nodeName:}" failed. No retries permitted until 2026-09-09 22:59:09.481656061 +0000 UTC m=+10.232240982 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pb2r" (UniqueName: "kubernetes.io/projected/b52163f6-1997-47c2-a86a-88a1cecaf4b4-kube-api-access-7pb2r") pod "network-check-target-mkgjx" (UID: "b52163f6-1997-47c2-a86a-88a1cecaf4b4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 22:59:05.848205 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:05.848171 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:05.848388 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:05.848316 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkgjx" podUID="b52163f6-1997-47c2-a86a-88a1cecaf4b4" Sep 09 22:59:05.848766 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:05.848740 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:05.848880 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:05.848852 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6l2s" podUID="55304a42-5933-48eb-bbbd-ed2f0d4b933b" Sep 09 22:59:05.848954 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:05.848921 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:05.849042 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:05.849020 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9h9s" podUID="185f1e13-d2f1-4c57-99d8-c3eb63e20512" Sep 09 22:59:07.849480 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:07.848537 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:07.849480 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:07.848630 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkgjx" podUID="b52163f6-1997-47c2-a86a-88a1cecaf4b4" Sep 09 22:59:07.849480 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:07.849097 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:07.849480 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:07.849204 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9h9s" podUID="185f1e13-d2f1-4c57-99d8-c3eb63e20512" Sep 09 22:59:07.849480 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:07.849260 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:07.849480 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:07.849331 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6l2s" podUID="55304a42-5933-48eb-bbbd-ed2f0d4b933b" Sep 09 22:59:09.415972 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:09.415820 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55304a42-5933-48eb-bbbd-ed2f0d4b933b-original-pull-secret\") pod \"global-pull-secret-syncer-b6l2s\" (UID: \"55304a42-5933-48eb-bbbd-ed2f0d4b933b\") " pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:09.415972 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:09.415886 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs\") pod \"network-metrics-daemon-l9h9s\" (UID: \"185f1e13-d2f1-4c57-99d8-c3eb63e20512\") " pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:09.416498 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:09.416009 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 22:59:09.416498 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:09.416074 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs podName:185f1e13-d2f1-4c57-99d8-c3eb63e20512 nodeName:}" failed. No retries permitted until 2026-09-09 22:59:17.41605464 +0000 UTC m=+18.166639560 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs") pod "network-metrics-daemon-l9h9s" (UID: "185f1e13-d2f1-4c57-99d8-c3eb63e20512") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 22:59:09.416618 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:09.416505 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 22:59:09.416618 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:09.416557 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55304a42-5933-48eb-bbbd-ed2f0d4b933b-original-pull-secret podName:55304a42-5933-48eb-bbbd-ed2f0d4b933b nodeName:}" failed. No retries permitted until 2026-09-09 22:59:17.416541686 +0000 UTC m=+18.167126614 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55304a42-5933-48eb-bbbd-ed2f0d4b933b-original-pull-secret") pod "global-pull-secret-syncer-b6l2s" (UID: "55304a42-5933-48eb-bbbd-ed2f0d4b933b") : object "kube-system"/"original-pull-secret" not registered Sep 09 22:59:09.517256 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:09.517214 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pb2r\" (UniqueName: \"kubernetes.io/projected/b52163f6-1997-47c2-a86a-88a1cecaf4b4-kube-api-access-7pb2r\") pod \"network-check-target-mkgjx\" (UID: \"b52163f6-1997-47c2-a86a-88a1cecaf4b4\") " pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:09.517462 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:09.517444 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 22:59:09.517537 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:09.517467 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 22:59:09.517537 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:09.517480 2578 projected.go:196] Error preparing data for projected volume kube-api-access-7pb2r for pod openshift-network-diagnostics/network-check-target-mkgjx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 22:59:09.517654 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:09.517550 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b52163f6-1997-47c2-a86a-88a1cecaf4b4-kube-api-access-7pb2r podName:b52163f6-1997-47c2-a86a-88a1cecaf4b4 nodeName:}" failed. No retries permitted until 2026-09-09 22:59:17.517529065 +0000 UTC m=+18.268113992 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pb2r" (UniqueName: "kubernetes.io/projected/b52163f6-1997-47c2-a86a-88a1cecaf4b4-kube-api-access-7pb2r") pod "network-check-target-mkgjx" (UID: "b52163f6-1997-47c2-a86a-88a1cecaf4b4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 22:59:09.847845 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:09.847135 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:09.847845 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:09.847238 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkgjx" podUID="b52163f6-1997-47c2-a86a-88a1cecaf4b4" Sep 09 22:59:09.847845 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:09.847589 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:09.847845 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:09.847680 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6l2s" podUID="55304a42-5933-48eb-bbbd-ed2f0d4b933b" Sep 09 22:59:09.847845 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:09.847733 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:09.847845 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:09.847810 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9h9s" podUID="185f1e13-d2f1-4c57-99d8-c3eb63e20512" Sep 09 22:59:11.846060 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:11.846025 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:11.846542 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:11.846035 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:11.846542 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:11.846163 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkgjx" podUID="b52163f6-1997-47c2-a86a-88a1cecaf4b4" Sep 09 22:59:11.846542 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:11.846035 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:11.846542 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:11.846252 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6l2s" podUID="55304a42-5933-48eb-bbbd-ed2f0d4b933b" Sep 09 22:59:11.846542 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:11.846331 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9h9s" podUID="185f1e13-d2f1-4c57-99d8-c3eb63e20512" Sep 09 22:59:13.846186 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:13.845940 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:13.846609 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:13.845940 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:13.846609 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:13.846289 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6l2s" podUID="55304a42-5933-48eb-bbbd-ed2f0d4b933b" Sep 09 22:59:13.846609 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:13.846334 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkgjx" podUID="b52163f6-1997-47c2-a86a-88a1cecaf4b4" Sep 09 22:59:13.846609 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:13.846000 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:13.846609 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:13.846533 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9h9s" podUID="185f1e13-d2f1-4c57-99d8-c3eb63e20512" Sep 09 22:59:15.846042 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:15.846002 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:15.846042 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:15.846025 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:15.846569 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:15.846138 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6l2s" podUID="55304a42-5933-48eb-bbbd-ed2f0d4b933b" Sep 09 22:59:15.846569 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:15.846504 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:15.846677 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:15.846606 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkgjx" podUID="b52163f6-1997-47c2-a86a-88a1cecaf4b4" Sep 09 22:59:15.846733 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:15.846713 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9h9s" podUID="185f1e13-d2f1-4c57-99d8-c3eb63e20512" Sep 09 22:59:17.474395 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:17.474353 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55304a42-5933-48eb-bbbd-ed2f0d4b933b-original-pull-secret\") pod \"global-pull-secret-syncer-b6l2s\" (UID: \"55304a42-5933-48eb-bbbd-ed2f0d4b933b\") " pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:17.474846 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:17.474419 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs\") pod \"network-metrics-daemon-l9h9s\" (UID: \"185f1e13-d2f1-4c57-99d8-c3eb63e20512\") " pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:17.474846 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:17.474511 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 22:59:17.474846 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:17.474511 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 22:59:17.474846 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:17.474573 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs podName:185f1e13-d2f1-4c57-99d8-c3eb63e20512 nodeName:}" failed. No retries permitted until 2026-09-09 22:59:33.474557972 +0000 UTC m=+34.225142876 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs") pod "network-metrics-daemon-l9h9s" (UID: "185f1e13-d2f1-4c57-99d8-c3eb63e20512") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 22:59:17.474846 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:17.474587 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55304a42-5933-48eb-bbbd-ed2f0d4b933b-original-pull-secret podName:55304a42-5933-48eb-bbbd-ed2f0d4b933b nodeName:}" failed. No retries permitted until 2026-09-09 22:59:33.474581272 +0000 UTC m=+34.225166177 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55304a42-5933-48eb-bbbd-ed2f0d4b933b-original-pull-secret") pod "global-pull-secret-syncer-b6l2s" (UID: "55304a42-5933-48eb-bbbd-ed2f0d4b933b") : object "kube-system"/"original-pull-secret" not registered Sep 09 22:59:17.575421 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:17.575395 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pb2r\" (UniqueName: \"kubernetes.io/projected/b52163f6-1997-47c2-a86a-88a1cecaf4b4-kube-api-access-7pb2r\") pod \"network-check-target-mkgjx\" (UID: \"b52163f6-1997-47c2-a86a-88a1cecaf4b4\") " pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:17.575571 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:17.575555 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 22:59:17.575610 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:17.575577 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 22:59:17.575610 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:17.575586 2578 projected.go:196] Error preparing data for projected volume kube-api-access-7pb2r for pod openshift-network-diagnostics/network-check-target-mkgjx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 22:59:17.575684 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:17.575636 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b52163f6-1997-47c2-a86a-88a1cecaf4b4-kube-api-access-7pb2r podName:b52163f6-1997-47c2-a86a-88a1cecaf4b4 nodeName:}" failed. No retries permitted until 2026-09-09 22:59:33.575623256 +0000 UTC m=+34.326208160 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pb2r" (UniqueName: "kubernetes.io/projected/b52163f6-1997-47c2-a86a-88a1cecaf4b4-kube-api-access-7pb2r") pod "network-check-target-mkgjx" (UID: "b52163f6-1997-47c2-a86a-88a1cecaf4b4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 22:59:17.846633 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:17.846593 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:17.846814 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:17.846644 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:17.846814 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:17.846593 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:17.846814 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:17.846735 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6l2s" podUID="55304a42-5933-48eb-bbbd-ed2f0d4b933b" Sep 09 22:59:17.846934 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:17.846822 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9h9s" podUID="185f1e13-d2f1-4c57-99d8-c3eb63e20512" Sep 09 22:59:17.846934 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:17.846912 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkgjx" podUID="b52163f6-1997-47c2-a86a-88a1cecaf4b4" Sep 09 22:59:19.847174 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:19.847136 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:19.847632 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:19.847255 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkgjx" podUID="b52163f6-1997-47c2-a86a-88a1cecaf4b4" Sep 09 22:59:19.847632 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:19.847332 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:19.847632 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:19.847454 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6l2s" podUID="55304a42-5933-48eb-bbbd-ed2f0d4b933b" Sep 09 22:59:19.847632 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:19.847501 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:19.847632 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:19.847581 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9h9s" podUID="185f1e13-d2f1-4c57-99d8-c3eb63e20512" Sep 09 22:59:21.846646 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:21.846422 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:21.847169 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:21.846718 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6l2s" podUID="55304a42-5933-48eb-bbbd-ed2f0d4b933b" Sep 09 22:59:21.847169 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:21.846440 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:21.847169 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:21.846800 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkgjx" podUID="b52163f6-1997-47c2-a86a-88a1cecaf4b4" Sep 09 22:59:21.847169 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:21.846440 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:21.847459 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:21.847328 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9h9s" podUID="185f1e13-d2f1-4c57-99d8-c3eb63e20512" Sep 09 22:59:21.970647 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:21.970624 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" event={"ID":"c3cbf093-0861-4893-8d8c-8ca3d30987c2","Type":"ContainerStarted","Data":"4fa37fbdb3a20798080723a7963b8740f8db87c1fb857241a316c1e5d2530470"} Sep 09 22:59:21.970735 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:21.970688 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" event={"ID":"c3cbf093-0861-4893-8d8c-8ca3d30987c2","Type":"ContainerStarted","Data":"d873c300d7ec23802620c0fc3cb759e93a00b61b36f0d7b5059e4739016845d6"} Sep 09 22:59:21.970735 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:21.970716 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" event={"ID":"c3cbf093-0861-4893-8d8c-8ca3d30987c2","Type":"ContainerStarted","Data":"11f6788e02f79134254b00c9c4ef203e3950d162d750874067ca2cb1d21d5a99"} Sep 09 22:59:21.970805 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:21.970744 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" event={"ID":"c3cbf093-0861-4893-8d8c-8ca3d30987c2","Type":"ContainerStarted","Data":"037340acba201051bde0b40086084624ee5bcd5219740471b9325578df9eb258"} Sep 09 22:59:21.971963 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:21.971936 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b8khc" event={"ID":"7f15552a-743f-44be-ae80-73ade41eba6f","Type":"ContainerStarted","Data":"863abbe2ac7cc807dc6069720437fd72e6ae68a7300f4849ec09f1f1a5397032"} Sep 09 22:59:21.973362 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:21.973338 2578 generic.go:356] "Generic (PLEG): container finished" podID="ec88f986-9c08-4e93-89c5-919467243b6f" containerID="a552e10150c2e5a69df066ea239f9b5c9665b17d06248507446071ad36eb6d60" exitCode=0 Sep 09 22:59:21.973448 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:21.973412 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kpcwm" event={"ID":"ec88f986-9c08-4e93-89c5-919467243b6f","Type":"ContainerDied","Data":"a552e10150c2e5a69df066ea239f9b5c9665b17d06248507446071ad36eb6d60"} Sep 09 22:59:21.974903 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:21.974887 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6j767" event={"ID":"6c9f70f3-174f-4bcf-9bb3-838910f4fce9","Type":"ContainerStarted","Data":"d4735b5d055964343eb05be3703cf1cdcca7675fcc392d65aecda65708191e8b"} Sep 09 22:59:21.976473 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:21.976452 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wc456" event={"ID":"3c974c39-606b-4f5d-ae55-cd15d6393611","Type":"ContainerStarted","Data":"b0dbfd4ecf8f73395ad2bfc42a1c97a0c8514a72f671317ff45342e025d6b8e8"} Sep 09 22:59:21.977759 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:21.977741 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-75rcr" event={"ID":"c864525a-b22a-4fef-ba7e-4b9541cbd717","Type":"ContainerStarted","Data":"781c31784287894077d1e2d4fdea93d1b43671e9e6b80ee6958914fd0ce8fc43"} Sep 09 22:59:21.982034 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:21.982015 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" event={"ID":"5e113c63-fae1-445e-872d-ca295524db02","Type":"ContainerStarted","Data":"44d3dce023667fd64b56f53149a40b21ef6d61c92e1d00ef00a3d68f9dbac04c"} Sep 09 22:59:21.983711 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:21.983687 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dsd8k" event={"ID":"b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d","Type":"ContainerStarted","Data":"5ed722c9aad745c3df4ac7b7e3655b88bf81220f5d391f0e2c0dbf560f3bbfeb"} Sep 09 22:59:21.988501 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:21.988468 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-b8khc" podStartSLOduration=3.674309741 podStartE2EDuration="21.988458015s" totalImagesPullingTime="18.314148274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 22:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:59:21.988325908 +0000 UTC m=+22.738910836" watchObservedRunningTime="2026-09-09 22:59:21.988458015 +0000 UTC m=+22.739042942" Sep 09 22:59:22.023821 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:22.023673 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6j767" podStartSLOduration=3.7275860400000003 podStartE2EDuration="22.023654926s" totalImagesPullingTime="18.296068886s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 22:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:59:22.022888441 +0000 UTC m=+22.773473387" watchObservedRunningTime="2026-09-09 22:59:22.023654926 +0000 UTC m=+22.774239857" Sep 09 22:59:22.036586 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:22.036545 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-75rcr" podStartSLOduration=3.755401383 podStartE2EDuration="22.036532687s" totalImagesPullingTime="18.281131304s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 22:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:59:22.036400848 +0000 UTC m=+22.786985806" watchObservedRunningTime="2026-09-09 22:59:22.036532687 +0000 UTC m=+22.787117614" Sep 09 22:59:22.050060 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:22.050019 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wc456" podStartSLOduration=7.273706932 podStartE2EDuration="22.050002312s" totalImagesPullingTime="14.77629538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:59:22.049647361 +0000 UTC m=+22.800232293" watchObservedRunningTime="2026-09-09 22:59:22.050002312 +0000 UTC m=+22.800587220" Sep 09 22:59:22.066812 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:22.066772 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dsd8k" podStartSLOduration=3.3999446779999998 podStartE2EDuration="22.066756141s" totalImagesPullingTime="18.666811463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 22:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:59:22.066021238 +0000 UTC m=+22.816606189" watchObservedRunningTime="2026-09-09 22:59:22.066756141 +0000 UTC m=+22.817341079" Sep 09 22:59:22.077316 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:22.077289 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wc456" Sep 09 22:59:22.077962 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:22.077942 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wc456" Sep 09 22:59:22.897041 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:22.897012 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 09 22:59:22.988440 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:22.988398 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" event={"ID":"5e113c63-fae1-445e-872d-ca295524db02","Type":"ContainerStarted","Data":"b2e87060100f573032262aea217adc53265bd7c3801df4870bed276363317ebf"} Sep 09 22:59:22.991972 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:22.991907 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" event={"ID":"c3cbf093-0861-4893-8d8c-8ca3d30987c2","Type":"ContainerStarted","Data":"fed35655dbb9ccdad0bab35f417f580ab7b20fa7f53c49c216c96213c0ce0262"} Sep 09 22:59:22.991972 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:22.991966 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" event={"ID":"c3cbf093-0861-4893-8d8c-8ca3d30987c2","Type":"ContainerStarted","Data":"3f03a5e8e7911bcfe37632cd230d85532f164cfaa4875628447da099aa98d12c"} Sep 09 22:59:22.993601 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:22.993575 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zbjsz" event={"ID":"df484634-6144-4f7a-bae1-3e7826c83e7b","Type":"ContainerStarted","Data":"542066d4dfe6ec17b450cb4f8d4f705d83b1bcf6809984f74bf964cdf514a455"} Sep 09 22:59:22.994344 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:22.994319 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wc456" Sep 09 22:59:22.994995 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:22.994978 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wc456" Sep 09 22:59:23.025182 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:23.025145 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zbjsz" podStartSLOduration=4.707764687 podStartE2EDuration="23.025129939s" totalImagesPullingTime="18.317365252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 22:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:59:23.010292048 +0000 UTC m=+23.760877001" watchObservedRunningTime="2026-09-09 22:59:23.025129939 +0000 UTC m=+23.775714868" Sep 09 22:59:23.807620 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:23.807513 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T22:59:22.897039903Z","UUID":"5dd6ca7e-57ae-46a3-b09e-87003a5fb1b2","Handler":null,"Name":"","Endpoint":""} Sep 09 22:59:23.809137 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:23.809114 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 09 22:59:23.809230 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:23.809152 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 09 22:59:23.846611 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:23.846584 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:23.846744 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:23.846585 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:23.846744 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:23.846700 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6l2s" podUID="55304a42-5933-48eb-bbbd-ed2f0d4b933b" Sep 09 22:59:23.846744 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:23.846714 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:23.846874 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:23.846791 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkgjx" podUID="b52163f6-1997-47c2-a86a-88a1cecaf4b4" Sep 09 22:59:23.846918 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:23.846884 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9h9s" podUID="185f1e13-d2f1-4c57-99d8-c3eb63e20512" Sep 09 22:59:23.998541 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:23.998497 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" event={"ID":"5e113c63-fae1-445e-872d-ca295524db02","Type":"ContainerStarted","Data":"850efc40029ef750c7c9fc2bdea95f4f0e42331227fedf5241c18e360bb6d14c"} Sep 09 22:59:24.019205 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:24.019146 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lhjb2" podStartSLOduration=3.737443021 podStartE2EDuration="24.019126799s" totalImagesPullingTime="20.281683778s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 22:59:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:59:24.018858442 +0000 UTC m=+24.769443391" watchObservedRunningTime="2026-09-09 22:59:24.019126799 +0000 UTC m=+24.769711727" Sep 09 22:59:25.005522 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:25.005248 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" event={"ID":"c3cbf093-0861-4893-8d8c-8ca3d30987c2","Type":"ContainerStarted","Data":"6416cbec68bad8ecdd50d65e9f77d870d2efbf997da01134ba490d7b020736e3"} Sep 09 22:59:25.846494 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:25.846465 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:25.846701 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:25.846465 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:25.846701 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:25.846590 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6l2s" podUID="55304a42-5933-48eb-bbbd-ed2f0d4b933b" Sep 09 22:59:25.846701 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:25.846469 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:25.846701 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:25.846657 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9h9s" podUID="185f1e13-d2f1-4c57-99d8-c3eb63e20512" Sep 09 22:59:25.846897 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:25.846746 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkgjx" podUID="b52163f6-1997-47c2-a86a-88a1cecaf4b4" Sep 09 22:59:27.014785 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:27.014603 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" event={"ID":"c3cbf093-0861-4893-8d8c-8ca3d30987c2","Type":"ContainerStarted","Data":"e9341c2636ad696041ab36f570f0a4209741be60f24f93794a413b4210ddd235"} Sep 09 22:59:27.015193 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:27.014865 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:27.015193 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:27.014924 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:27.015193 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:27.014975 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:27.016886 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:27.016854 2578 generic.go:356] "Generic (PLEG): container finished" podID="ec88f986-9c08-4e93-89c5-919467243b6f" containerID="66e0d7e0f8deb6bdca905b0a1549b1dfaa1d687a7927659176e2f4446767cffb" exitCode=0 Sep 09 22:59:27.017011 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:27.016892 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kpcwm" event={"ID":"ec88f986-9c08-4e93-89c5-919467243b6f","Type":"ContainerDied","Data":"66e0d7e0f8deb6bdca905b0a1549b1dfaa1d687a7927659176e2f4446767cffb"} Sep 09 22:59:27.031201 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:27.031181 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:27.031861 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:27.031806 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 22:59:27.041370 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:27.041332 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" podStartSLOduration=8.40247157 podStartE2EDuration="27.041319937s" totalImagesPullingTime="18.638848367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 22:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:59:27.04076158 +0000 UTC m=+27.791346537" watchObservedRunningTime="2026-09-09 22:59:27.041319937 +0000 UTC m=+27.791904864" Sep 09 22:59:27.845653 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:27.845624 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:27.845653 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:27.845646 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:27.845939 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:27.845628 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:27.845939 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:27.845739 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkgjx" podUID="b52163f6-1997-47c2-a86a-88a1cecaf4b4" Sep 09 22:59:27.845939 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:27.845815 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6l2s" podUID="55304a42-5933-48eb-bbbd-ed2f0d4b933b" Sep 09 22:59:27.845939 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:27.845914 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9h9s" podUID="185f1e13-d2f1-4c57-99d8-c3eb63e20512" Sep 09 22:59:28.304330 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:28.304297 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b6l2s"] Sep 09 22:59:28.305137 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:28.304472 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:28.305137 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:28.304570 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6l2s" podUID="55304a42-5933-48eb-bbbd-ed2f0d4b933b" Sep 09 22:59:28.305137 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:28.305063 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mkgjx"] Sep 09 22:59:28.305137 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:28.305137 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:28.305343 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:28.305205 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkgjx" podUID="b52163f6-1997-47c2-a86a-88a1cecaf4b4" Sep 09 22:59:28.315518 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:28.315498 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l9h9s"] Sep 09 22:59:28.315623 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:28.315585 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:28.315681 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:28.315665 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9h9s" podUID="185f1e13-d2f1-4c57-99d8-c3eb63e20512" Sep 09 22:59:29.023908 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:29.023874 2578 generic.go:356] "Generic (PLEG): container finished" podID="ec88f986-9c08-4e93-89c5-919467243b6f" containerID="e2e7b7335f1118f90772d5f19b9839a441b3f1c1c475f6c475526ba5e585e6df" exitCode=0 Sep 09 22:59:29.024113 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:29.023953 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kpcwm" event={"ID":"ec88f986-9c08-4e93-89c5-919467243b6f","Type":"ContainerDied","Data":"e2e7b7335f1118f90772d5f19b9839a441b3f1c1c475f6c475526ba5e585e6df"} Sep 09 22:59:29.847190 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:29.847159 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:29.847550 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:29.847251 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkgjx" podUID="b52163f6-1997-47c2-a86a-88a1cecaf4b4" Sep 09 22:59:29.847550 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:29.847358 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:29.847550 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:29.847498 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6l2s" podUID="55304a42-5933-48eb-bbbd-ed2f0d4b933b" Sep 09 22:59:29.847550 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:29.847535 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:29.847696 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:29.847626 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9h9s" podUID="185f1e13-d2f1-4c57-99d8-c3eb63e20512" Sep 09 22:59:31.031513 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:31.031298 2578 generic.go:356] "Generic (PLEG): container finished" podID="ec88f986-9c08-4e93-89c5-919467243b6f" containerID="37749737c31164ab5afce848c40925507446eece90b2ea19d997cb9daf714bc4" exitCode=0 Sep 09 22:59:31.031513 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:31.031397 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kpcwm" event={"ID":"ec88f986-9c08-4e93-89c5-919467243b6f","Type":"ContainerDied","Data":"37749737c31164ab5afce848c40925507446eece90b2ea19d997cb9daf714bc4"} Sep 09 22:59:31.846323 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:31.846282 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:31.846547 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:31.846290 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:31.846547 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:31.846440 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkgjx" podUID="b52163f6-1997-47c2-a86a-88a1cecaf4b4" Sep 09 22:59:31.846547 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:31.846476 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6l2s" podUID="55304a42-5933-48eb-bbbd-ed2f0d4b933b" Sep 09 22:59:31.846547 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:31.846290 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:31.846750 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:31.846579 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9h9s" podUID="185f1e13-d2f1-4c57-99d8-c3eb63e20512" Sep 09 22:59:33.486910 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:33.486878 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs\") pod \"network-metrics-daemon-l9h9s\" (UID: \"185f1e13-d2f1-4c57-99d8-c3eb63e20512\") " pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:33.487494 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:33.486961 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55304a42-5933-48eb-bbbd-ed2f0d4b933b-original-pull-secret\") pod \"global-pull-secret-syncer-b6l2s\" (UID: \"55304a42-5933-48eb-bbbd-ed2f0d4b933b\") " pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:33.487494 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:33.487055 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 22:59:33.487494 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:33.487064 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 22:59:33.487494 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:33.487121 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55304a42-5933-48eb-bbbd-ed2f0d4b933b-original-pull-secret podName:55304a42-5933-48eb-bbbd-ed2f0d4b933b nodeName:}" failed. No retries permitted until 2026-09-09 23:00:05.487104413 +0000 UTC m=+66.237689320 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55304a42-5933-48eb-bbbd-ed2f0d4b933b-original-pull-secret") pod "global-pull-secret-syncer-b6l2s" (UID: "55304a42-5933-48eb-bbbd-ed2f0d4b933b") : object "kube-system"/"original-pull-secret" not registered Sep 09 22:59:33.487494 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:33.487135 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs podName:185f1e13-d2f1-4c57-99d8-c3eb63e20512 nodeName:}" failed. No retries permitted until 2026-09-09 23:00:05.487128778 +0000 UTC m=+66.237713683 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs") pod "network-metrics-daemon-l9h9s" (UID: "185f1e13-d2f1-4c57-99d8-c3eb63e20512") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 22:59:33.588339 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:33.588294 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pb2r\" (UniqueName: \"kubernetes.io/projected/b52163f6-1997-47c2-a86a-88a1cecaf4b4-kube-api-access-7pb2r\") pod \"network-check-target-mkgjx\" (UID: \"b52163f6-1997-47c2-a86a-88a1cecaf4b4\") " pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:33.588535 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:33.588470 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 22:59:33.588535 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:33.588491 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 22:59:33.588535 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:33.588501 2578 projected.go:196] Error preparing data for projected volume kube-api-access-7pb2r for pod openshift-network-diagnostics/network-check-target-mkgjx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 22:59:33.588707 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:33.588563 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b52163f6-1997-47c2-a86a-88a1cecaf4b4-kube-api-access-7pb2r podName:b52163f6-1997-47c2-a86a-88a1cecaf4b4 nodeName:}" failed. No retries permitted until 2026-09-09 23:00:05.58854797 +0000 UTC m=+66.339132900 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pb2r" (UniqueName: "kubernetes.io/projected/b52163f6-1997-47c2-a86a-88a1cecaf4b4-kube-api-access-7pb2r") pod "network-check-target-mkgjx" (UID: "b52163f6-1997-47c2-a86a-88a1cecaf4b4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 22:59:33.845787 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:33.845745 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:33.845972 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:33.845745 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:33.845972 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:33.845860 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mkgjx" podUID="b52163f6-1997-47c2-a86a-88a1cecaf4b4" Sep 09 22:59:33.845972 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:33.845916 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b6l2s" podUID="55304a42-5933-48eb-bbbd-ed2f0d4b933b" Sep 09 22:59:33.845972 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:33.845759 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:33.846198 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:33.846046 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9h9s" podUID="185f1e13-d2f1-4c57-99d8-c3eb63e20512" Sep 09 22:59:34.037483 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.037453 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-252.ec2.internal" event="NodeReady" Sep 09 22:59:34.037663 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.037646 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 09 22:59:34.078512 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.078448 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wwkz5"] Sep 09 22:59:34.084751 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.084719 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9prb4"] Sep 09 22:59:34.084914 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.084802 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wwkz5" Sep 09 22:59:34.086598 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.086578 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9prb4" Sep 09 22:59:34.087132 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.087111 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 09 22:59:34.087228 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.087150 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t7bbt\"" Sep 09 22:59:34.087228 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.087120 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 09 22:59:34.088790 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.088771 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 09 22:59:34.089553 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.089532 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dx997\"" Sep 09 22:59:34.089779 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.089763 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 09 22:59:34.090051 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.090032 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 09 22:59:34.090839 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.090779 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9prb4"] Sep 09 22:59:34.091809 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.091791 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wwkz5"] Sep 09 22:59:34.193808 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.193779 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-config-volume\") pod \"dns-default-wwkz5\" (UID: \"e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec\") " pod="openshift-dns/dns-default-wwkz5" Sep 09 22:59:34.194003 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.193819 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nktct\" (UniqueName: \"kubernetes.io/projected/7d14c654-5bd8-4b10-aef1-32bf396708ab-kube-api-access-nktct\") pod \"ingress-canary-9prb4\" (UID: \"7d14c654-5bd8-4b10-aef1-32bf396708ab\") " pod="openshift-ingress-canary/ingress-canary-9prb4" Sep 09 22:59:34.194003 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.193844 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbpjw\" (UniqueName: \"kubernetes.io/projected/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-kube-api-access-dbpjw\") pod \"dns-default-wwkz5\" (UID: \"e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec\") " pod="openshift-dns/dns-default-wwkz5" Sep 09 22:59:34.194003 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.193895 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d14c654-5bd8-4b10-aef1-32bf396708ab-cert\") pod \"ingress-canary-9prb4\" (UID: \"7d14c654-5bd8-4b10-aef1-32bf396708ab\") " pod="openshift-ingress-canary/ingress-canary-9prb4" Sep 09 22:59:34.194003 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.193971 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-tmp-dir\") pod \"dns-default-wwkz5\" (UID: \"e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec\") " pod="openshift-dns/dns-default-wwkz5" Sep 09 22:59:34.194180 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.194047 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls\") pod \"dns-default-wwkz5\" (UID: \"e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec\") " pod="openshift-dns/dns-default-wwkz5" Sep 09 22:59:34.295287 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.295250 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls\") pod \"dns-default-wwkz5\" (UID: \"e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec\") " pod="openshift-dns/dns-default-wwkz5" Sep 09 22:59:34.295484 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.295312 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-config-volume\") pod \"dns-default-wwkz5\" (UID: \"e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec\") " pod="openshift-dns/dns-default-wwkz5" Sep 09 22:59:34.295484 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.295342 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nktct\" (UniqueName: \"kubernetes.io/projected/7d14c654-5bd8-4b10-aef1-32bf396708ab-kube-api-access-nktct\") pod \"ingress-canary-9prb4\" (UID: \"7d14c654-5bd8-4b10-aef1-32bf396708ab\") " pod="openshift-ingress-canary/ingress-canary-9prb4" Sep 09 22:59:34.295484 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.295367 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbpjw\" (UniqueName: \"kubernetes.io/projected/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-kube-api-access-dbpjw\") pod \"dns-default-wwkz5\" (UID: \"e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec\") " pod="openshift-dns/dns-default-wwkz5" Sep 09 22:59:34.295484 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.295410 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d14c654-5bd8-4b10-aef1-32bf396708ab-cert\") pod \"ingress-canary-9prb4\" (UID: \"7d14c654-5bd8-4b10-aef1-32bf396708ab\") " pod="openshift-ingress-canary/ingress-canary-9prb4" Sep 09 22:59:34.295484 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:34.295424 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 22:59:34.295484 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.295439 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-tmp-dir\") pod \"dns-default-wwkz5\" (UID: \"e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec\") " pod="openshift-dns/dns-default-wwkz5" Sep 09 22:59:34.295484 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:34.295486 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls podName:e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec nodeName:}" failed. No retries permitted until 2026-09-09 22:59:34.795468119 +0000 UTC m=+35.546053024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls") pod "dns-default-wwkz5" (UID: "e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec") : secret "dns-default-metrics-tls" not found Sep 09 22:59:34.295863 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.295836 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-tmp-dir\") pod \"dns-default-wwkz5\" (UID: \"e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec\") " pod="openshift-dns/dns-default-wwkz5" Sep 09 22:59:34.296485 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.296459 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-config-volume\") pod \"dns-default-wwkz5\" (UID: \"e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec\") " pod="openshift-dns/dns-default-wwkz5" Sep 09 22:59:34.300994 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.300976 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d14c654-5bd8-4b10-aef1-32bf396708ab-cert\") pod \"ingress-canary-9prb4\" (UID: \"7d14c654-5bd8-4b10-aef1-32bf396708ab\") " pod="openshift-ingress-canary/ingress-canary-9prb4" Sep 09 22:59:34.304315 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.304296 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nktct\" (UniqueName: \"kubernetes.io/projected/7d14c654-5bd8-4b10-aef1-32bf396708ab-kube-api-access-nktct\") pod \"ingress-canary-9prb4\" (UID: \"7d14c654-5bd8-4b10-aef1-32bf396708ab\") " pod="openshift-ingress-canary/ingress-canary-9prb4" Sep 09 22:59:34.304443 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.304405 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbpjw\" (UniqueName: \"kubernetes.io/projected/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-kube-api-access-dbpjw\") pod \"dns-default-wwkz5\" (UID: \"e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec\") " pod="openshift-dns/dns-default-wwkz5" Sep 09 22:59:34.402659 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.402624 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9prb4" Sep 09 22:59:34.800869 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:34.800833 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls\") pod \"dns-default-wwkz5\" (UID: \"e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec\") " pod="openshift-dns/dns-default-wwkz5" Sep 09 22:59:34.801296 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:34.801049 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 22:59:34.801296 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:34.801131 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls podName:e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec nodeName:}" failed. No retries permitted until 2026-09-09 22:59:35.80110873 +0000 UTC m=+36.551693641 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls") pod "dns-default-wwkz5" (UID: "e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec") : secret "dns-default-metrics-tls" not found Sep 09 22:59:35.182786 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:35.182601 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9prb4"] Sep 09 22:59:35.186186 ip-10-0-139-252 kubenswrapper[2578]: W0909 22:59:35.186160 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d14c654_5bd8_4b10_aef1_32bf396708ab.slice/crio-be690bff03248dce3ebf17bb0a19f83a7d4ccbddaa135a27146016fd5e7db4cf WatchSource:0}: Error finding container be690bff03248dce3ebf17bb0a19f83a7d4ccbddaa135a27146016fd5e7db4cf: Status 404 returned error can't find the container with id be690bff03248dce3ebf17bb0a19f83a7d4ccbddaa135a27146016fd5e7db4cf Sep 09 22:59:35.809590 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:35.809553 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls\") pod \"dns-default-wwkz5\" (UID: \"e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec\") " pod="openshift-dns/dns-default-wwkz5" Sep 09 22:59:35.810105 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:35.809718 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 22:59:35.810105 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:35.809792 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls podName:e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec nodeName:}" failed. No retries permitted until 2026-09-09 22:59:37.809765437 +0000 UTC m=+38.560350342 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls") pod "dns-default-wwkz5" (UID: "e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec") : secret "dns-default-metrics-tls" not found Sep 09 22:59:35.846084 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:35.846052 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 22:59:35.846084 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:35.846085 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 22:59:35.846328 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:35.846052 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 22:59:35.848613 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:35.848593 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 22:59:35.849915 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:35.849633 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 22:59:35.849915 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:35.849647 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sns2d\"" Sep 09 22:59:35.849915 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:35.849676 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 22:59:35.849915 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:35.849698 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vjfnz\"" Sep 09 22:59:35.849915 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:35.849708 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 22:59:36.047130 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:36.047097 2578 generic.go:356] "Generic (PLEG): container finished" podID="ec88f986-9c08-4e93-89c5-919467243b6f" containerID="1ac90e3b74d1aab0d6817d61afe294bb3598eff37b28508d484eb568781014e0" exitCode=0 Sep 09 22:59:36.047299 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:36.047186 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kpcwm" event={"ID":"ec88f986-9c08-4e93-89c5-919467243b6f","Type":"ContainerDied","Data":"1ac90e3b74d1aab0d6817d61afe294bb3598eff37b28508d484eb568781014e0"} Sep 09 22:59:36.048739 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:36.048703 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9prb4" event={"ID":"7d14c654-5bd8-4b10-aef1-32bf396708ab","Type":"ContainerStarted","Data":"be690bff03248dce3ebf17bb0a19f83a7d4ccbddaa135a27146016fd5e7db4cf"} Sep 09 22:59:37.058630 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:37.058578 2578 generic.go:356] "Generic (PLEG): container finished" podID="ec88f986-9c08-4e93-89c5-919467243b6f" containerID="b4730123a1eeb4f75a10efd9d2a62c1a52d587ede83f9ee87680ab068d8947e3" exitCode=0 Sep 09 22:59:37.059279 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:37.058659 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kpcwm" event={"ID":"ec88f986-9c08-4e93-89c5-919467243b6f","Type":"ContainerDied","Data":"b4730123a1eeb4f75a10efd9d2a62c1a52d587ede83f9ee87680ab068d8947e3"} Sep 09 22:59:37.825868 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:37.825828 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls\") pod \"dns-default-wwkz5\" (UID: \"e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec\") " pod="openshift-dns/dns-default-wwkz5" Sep 09 22:59:37.826081 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:37.826062 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 22:59:37.826172 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:37.826126 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls podName:e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec nodeName:}" failed. No retries permitted until 2026-09-09 22:59:41.82610816 +0000 UTC m=+42.576693080 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls") pod "dns-default-wwkz5" (UID: "e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec") : secret "dns-default-metrics-tls" not found Sep 09 22:59:38.064643 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:38.064610 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kpcwm" event={"ID":"ec88f986-9c08-4e93-89c5-919467243b6f","Type":"ContainerStarted","Data":"c4a4cb8a94383ae0ba22cc85cef3592508ff6c1f03e748ba4b0b54d40fbf4644"} Sep 09 22:59:38.066088 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:38.066063 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9prb4" event={"ID":"7d14c654-5bd8-4b10-aef1-32bf396708ab","Type":"ContainerStarted","Data":"94ad82f8e6d417cd448e0655d1e3323ee700350a342a8c077d6741172d36dd1b"} Sep 09 22:59:38.085772 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:38.085732 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kpcwm" podStartSLOduration=9.479453659 podStartE2EDuration="38.085719319s" totalImagesPullingTime="28.60626566s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 22:59:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:59:38.084433061 +0000 UTC m=+38.835018007" watchObservedRunningTime="2026-09-09 22:59:38.085719319 +0000 UTC m=+38.836304246" Sep 09 22:59:38.099602 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:38.099564 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9prb4" podStartSLOduration=1.706618133 podStartE2EDuration="4.099550394s" totalImagesPullingTime="2.392932261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:59:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 22:59:38.098828699 +0000 UTC m=+38.849413649" watchObservedRunningTime="2026-09-09 22:59:38.099550394 +0000 UTC m=+38.850135321" Sep 09 22:59:41.854193 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:41.854007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls\") pod \"dns-default-wwkz5\" (UID: \"e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec\") " pod="openshift-dns/dns-default-wwkz5" Sep 09 22:59:41.854193 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:41.854154 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 22:59:41.854631 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:41.854242 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls podName:e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec nodeName:}" failed. No retries permitted until 2026-09-09 22:59:49.854227629 +0000 UTC m=+50.604812535 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls") pod "dns-default-wwkz5" (UID: "e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec") : secret "dns-default-metrics-tls" not found Sep 09 22:59:49.908472 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:49.908395 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls\") pod \"dns-default-wwkz5\" (UID: \"e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec\") " pod="openshift-dns/dns-default-wwkz5" Sep 09 22:59:49.908838 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:49.908472 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 22:59:49.908838 ip-10-0-139-252 kubenswrapper[2578]: E0909 22:59:49.908540 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls podName:e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec nodeName:}" failed. No retries permitted until 2026-09-09 23:00:05.908523484 +0000 UTC m=+66.659108389 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls") pod "dns-default-wwkz5" (UID: "e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec") : secret "dns-default-metrics-tls" not found Sep 09 22:59:59.044666 ip-10-0-139-252 kubenswrapper[2578]: I0909 22:59:59.044637 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4vtjn" Sep 09 23:00:05.517489 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:05.517447 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs\") pod \"network-metrics-daemon-l9h9s\" (UID: \"185f1e13-d2f1-4c57-99d8-c3eb63e20512\") " pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 23:00:05.517963 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:05.517519 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55304a42-5933-48eb-bbbd-ed2f0d4b933b-original-pull-secret\") pod \"global-pull-secret-syncer-b6l2s\" (UID: \"55304a42-5933-48eb-bbbd-ed2f0d4b933b\") " pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 23:00:05.520245 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:05.520223 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 23:00:05.520338 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:05.520262 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 23:00:05.528331 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:00:05.528311 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 23:00:05.528431 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:00:05.528410 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs podName:185f1e13-d2f1-4c57-99d8-c3eb63e20512 nodeName:}" failed. No retries permitted until 2026-09-09 23:01:09.528369624 +0000 UTC m=+130.278954529 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs") pod "network-metrics-daemon-l9h9s" (UID: "185f1e13-d2f1-4c57-99d8-c3eb63e20512") : secret "metrics-daemon-secret" not found Sep 09 23:00:05.531458 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:05.531427 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55304a42-5933-48eb-bbbd-ed2f0d4b933b-original-pull-secret\") pod \"global-pull-secret-syncer-b6l2s\" (UID: \"55304a42-5933-48eb-bbbd-ed2f0d4b933b\") " pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 23:00:05.557487 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:05.557461 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b6l2s" Sep 09 23:00:05.618761 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:05.618723 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pb2r\" (UniqueName: \"kubernetes.io/projected/b52163f6-1997-47c2-a86a-88a1cecaf4b4-kube-api-access-7pb2r\") pod \"network-check-target-mkgjx\" (UID: \"b52163f6-1997-47c2-a86a-88a1cecaf4b4\") " pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 23:00:05.621477 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:05.621453 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 23:00:05.631571 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:05.631547 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 23:00:05.642374 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:05.642350 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pb2r\" (UniqueName: \"kubernetes.io/projected/b52163f6-1997-47c2-a86a-88a1cecaf4b4-kube-api-access-7pb2r\") pod \"network-check-target-mkgjx\" (UID: \"b52163f6-1997-47c2-a86a-88a1cecaf4b4\") " pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 23:00:05.850696 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:05.850668 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b6l2s"] Sep 09 23:00:05.851440 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:00:05.851413 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55304a42_5933_48eb_bbbd_ed2f0d4b933b.slice/crio-a42807efe1cbd9c81ff823efa60a280f7cd3f6dd50a711779742556ca9ac20d9 WatchSource:0}: Error finding container a42807efe1cbd9c81ff823efa60a280f7cd3f6dd50a711779742556ca9ac20d9: Status 404 returned error can't find the container with id a42807efe1cbd9c81ff823efa60a280f7cd3f6dd50a711779742556ca9ac20d9 Sep 09 23:00:05.866703 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:05.866683 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sns2d\"" Sep 09 23:00:05.874759 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:05.874742 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 23:00:05.921259 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:05.921235 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls\") pod \"dns-default-wwkz5\" (UID: \"e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec\") " pod="openshift-dns/dns-default-wwkz5" Sep 09 23:00:05.921398 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:00:05.921367 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:00:05.921466 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:00:05.921441 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls podName:e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec nodeName:}" failed. No retries permitted until 2026-09-09 23:00:37.921425263 +0000 UTC m=+98.672010187 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls") pod "dns-default-wwkz5" (UID: "e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec") : secret "dns-default-metrics-tls" not found Sep 09 23:00:06.129717 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:06.129631 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b6l2s" event={"ID":"55304a42-5933-48eb-bbbd-ed2f0d4b933b","Type":"ContainerStarted","Data":"a42807efe1cbd9c81ff823efa60a280f7cd3f6dd50a711779742556ca9ac20d9"} Sep 09 23:00:06.176569 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:06.176543 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mkgjx"] Sep 09 23:00:06.179337 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:00:06.179314 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb52163f6_1997_47c2_a86a_88a1cecaf4b4.slice/crio-4c0f54931e10a2a404bca2bd7d5d9b96e7ead9de953c332fa98bcfcf3cbc05bc WatchSource:0}: Error finding container 4c0f54931e10a2a404bca2bd7d5d9b96e7ead9de953c332fa98bcfcf3cbc05bc: Status 404 returned error can't find the container with id 4c0f54931e10a2a404bca2bd7d5d9b96e7ead9de953c332fa98bcfcf3cbc05bc Sep 09 23:00:07.133717 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:07.133649 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mkgjx" event={"ID":"b52163f6-1997-47c2-a86a-88a1cecaf4b4","Type":"ContainerStarted","Data":"4c0f54931e10a2a404bca2bd7d5d9b96e7ead9de953c332fa98bcfcf3cbc05bc"} Sep 09 23:00:11.146727 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:11.146689 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b6l2s" event={"ID":"55304a42-5933-48eb-bbbd-ed2f0d4b933b","Type":"ContainerStarted","Data":"b2c9979ef8ce6def8e12571d0d50cdb34346e599f463a79cd6ea9f5d8e52d97d"} Sep 09 23:00:11.148308 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:11.148275 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mkgjx" event={"ID":"b52163f6-1997-47c2-a86a-88a1cecaf4b4","Type":"ContainerStarted","Data":"a657587f65805b5eb565df2906f32916bd8bec5652aaa518918b33b0b006d148"} Sep 09 23:00:11.148552 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:11.148430 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 23:00:11.162305 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:11.162254 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-b6l2s" podStartSLOduration=66.076260612 podStartE2EDuration="1m11.162237613s" totalImagesPullingTime="5.085977001s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 22:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:00:11.161580079 +0000 UTC m=+71.912164987" watchObservedRunningTime="2026-09-09 23:00:11.162237613 +0000 UTC m=+71.912822540" Sep 09 23:00:11.175516 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:11.175473 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mkgjx" podStartSLOduration=66.417729232 podStartE2EDuration="1m11.175460284s" totalImagesPullingTime="4.757731052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:00:11.174574976 +0000 UTC m=+71.925159918" watchObservedRunningTime="2026-09-09 23:00:11.175460284 +0000 UTC m=+71.926045211" Sep 09 23:00:30.316395 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:30.316262 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776587dd6b-bkb8g"] Sep 09 23:00:30.320424 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:30.320397 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776587dd6b-bkb8g" Sep 09 23:00:30.323959 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:30.323937 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-x295c\"" Sep 09 23:00:30.324064 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:30.323938 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 09 23:00:30.324064 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:30.323939 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 09 23:00:30.324064 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:30.323940 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 09 23:00:30.324064 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:30.324006 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 09 23:00:30.329150 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:30.329128 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776587dd6b-bkb8g"] Sep 09 23:00:30.392856 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:30.392828 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/992f05ce-c6ce-4874-ba8b-51efef552e61-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-776587dd6b-bkb8g\" (UID: \"992f05ce-c6ce-4874-ba8b-51efef552e61\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776587dd6b-bkb8g" Sep 09 23:00:30.392986 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:30.392877 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsq7s\" (UniqueName: \"kubernetes.io/projected/992f05ce-c6ce-4874-ba8b-51efef552e61-kube-api-access-lsq7s\") pod \"managed-serviceaccount-addon-agent-776587dd6b-bkb8g\" (UID: \"992f05ce-c6ce-4874-ba8b-51efef552e61\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776587dd6b-bkb8g" Sep 09 23:00:30.493270 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:30.493246 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsq7s\" (UniqueName: \"kubernetes.io/projected/992f05ce-c6ce-4874-ba8b-51efef552e61-kube-api-access-lsq7s\") pod \"managed-serviceaccount-addon-agent-776587dd6b-bkb8g\" (UID: \"992f05ce-c6ce-4874-ba8b-51efef552e61\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776587dd6b-bkb8g" Sep 09 23:00:30.493419 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:30.493315 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/992f05ce-c6ce-4874-ba8b-51efef552e61-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-776587dd6b-bkb8g\" (UID: \"992f05ce-c6ce-4874-ba8b-51efef552e61\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776587dd6b-bkb8g" Sep 09 23:00:30.495879 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:30.495860 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/992f05ce-c6ce-4874-ba8b-51efef552e61-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-776587dd6b-bkb8g\" (UID: \"992f05ce-c6ce-4874-ba8b-51efef552e61\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776587dd6b-bkb8g" Sep 09 23:00:30.500200 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:30.500180 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsq7s\" (UniqueName: \"kubernetes.io/projected/992f05ce-c6ce-4874-ba8b-51efef552e61-kube-api-access-lsq7s\") pod \"managed-serviceaccount-addon-agent-776587dd6b-bkb8g\" (UID: \"992f05ce-c6ce-4874-ba8b-51efef552e61\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776587dd6b-bkb8g" Sep 09 23:00:30.643154 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:30.643091 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776587dd6b-bkb8g" Sep 09 23:00:30.908349 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:30.908280 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776587dd6b-bkb8g"] Sep 09 23:00:30.911492 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:00:30.911465 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod992f05ce_c6ce_4874_ba8b_51efef552e61.slice/crio-abe449e864637a95a4714c07ca8ac868c891f99c933862a4b81851ba35b25ab7 WatchSource:0}: Error finding container abe449e864637a95a4714c07ca8ac868c891f99c933862a4b81851ba35b25ab7: Status 404 returned error can't find the container with id abe449e864637a95a4714c07ca8ac868c891f99c933862a4b81851ba35b25ab7 Sep 09 23:00:31.197245 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:31.197167 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776587dd6b-bkb8g" event={"ID":"992f05ce-c6ce-4874-ba8b-51efef552e61","Type":"ContainerStarted","Data":"abe449e864637a95a4714c07ca8ac868c891f99c933862a4b81851ba35b25ab7"} Sep 09 23:00:34.206198 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:34.206160 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776587dd6b-bkb8g" event={"ID":"992f05ce-c6ce-4874-ba8b-51efef552e61","Type":"ContainerStarted","Data":"603545c642a14b4a72e10df1ad978840e053d2224b25d74d4fd0a866e08e1afb"} Sep 09 23:00:34.222687 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:34.222640 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776587dd6b-bkb8g" podStartSLOduration=1.629444611 podStartE2EDuration="4.222625332s" totalImagesPullingTime="2.593180721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:00:34.222056208 +0000 UTC m=+94.972641137" watchObservedRunningTime="2026-09-09 23:00:34.222625332 +0000 UTC m=+94.973210280" Sep 09 23:00:37.945733 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:37.945691 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls\") pod \"dns-default-wwkz5\" (UID: \"e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec\") " pod="openshift-dns/dns-default-wwkz5" Sep 09 23:00:37.946066 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:00:37.945849 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:00:37.946066 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:00:37.945940 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls podName:e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec nodeName:}" failed. No retries permitted until 2026-09-09 23:01:41.945920953 +0000 UTC m=+162.696505859 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls") pod "dns-default-wwkz5" (UID: "e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec") : secret "dns-default-metrics-tls" not found Sep 09 23:00:42.153232 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:00:42.153203 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mkgjx" Sep 09 23:01:09.553750 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:01:09.553688 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs\") pod \"network-metrics-daemon-l9h9s\" (UID: \"185f1e13-d2f1-4c57-99d8-c3eb63e20512\") " pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 23:01:09.554252 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:01:09.553858 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 23:01:09.554252 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:01:09.553945 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs podName:185f1e13-d2f1-4c57-99d8-c3eb63e20512 nodeName:}" failed. No retries permitted until 2026-09-09 23:03:11.553926862 +0000 UTC m=+252.304511768 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs") pod "network-metrics-daemon-l9h9s" (UID: "185f1e13-d2f1-4c57-99d8-c3eb63e20512") : secret "metrics-daemon-secret" not found Sep 09 23:01:37.097591 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:01:37.097538 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-wwkz5" podUID="e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec" Sep 09 23:01:37.350703 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:01:37.350627 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wwkz5" Sep 09 23:01:38.869827 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:01:38.869785 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-l9h9s" podUID="185f1e13-d2f1-4c57-99d8-c3eb63e20512" Sep 09 23:01:41.957395 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:01:41.957348 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls\") pod \"dns-default-wwkz5\" (UID: \"e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec\") " pod="openshift-dns/dns-default-wwkz5" Sep 09 23:01:41.957851 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:01:41.957490 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:01:41.957851 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:01:41.957563 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls podName:e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec nodeName:}" failed. No retries permitted until 2026-09-09 23:03:43.957547055 +0000 UTC m=+284.708131960 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls") pod "dns-default-wwkz5" (UID: "e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec") : secret "dns-default-metrics-tls" not found Sep 09 23:01:50.846138 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:01:50.846032 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 23:02:34.488982 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:02:34.488943 2578 generic.go:356] "Generic (PLEG): container finished" podID="992f05ce-c6ce-4874-ba8b-51efef552e61" containerID="603545c642a14b4a72e10df1ad978840e053d2224b25d74d4fd0a866e08e1afb" exitCode=255 Sep 09 23:02:34.489402 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:02:34.488997 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776587dd6b-bkb8g" event={"ID":"992f05ce-c6ce-4874-ba8b-51efef552e61","Type":"ContainerDied","Data":"603545c642a14b4a72e10df1ad978840e053d2224b25d74d4fd0a866e08e1afb"} Sep 09 23:02:34.494277 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:02:34.494257 2578 scope.go:117] "RemoveContainer" containerID="603545c642a14b4a72e10df1ad978840e053d2224b25d74d4fd0a866e08e1afb" Sep 09 23:02:35.492973 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:02:35.492940 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776587dd6b-bkb8g" event={"ID":"992f05ce-c6ce-4874-ba8b-51efef552e61","Type":"ContainerStarted","Data":"5e9a259651bb3a90e3e93715d749df397d67523a2dadd171ae4206c32e9b4548"} Sep 09 23:03:11.644578 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:11.644541 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs\") pod \"network-metrics-daemon-l9h9s\" (UID: \"185f1e13-d2f1-4c57-99d8-c3eb63e20512\") " pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 23:03:11.645063 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:11.644688 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 23:03:11.645063 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:11.644745 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs podName:185f1e13-d2f1-4c57-99d8-c3eb63e20512 nodeName:}" failed. No retries permitted until 2026-09-09 23:05:13.644729646 +0000 UTC m=+374.395314551 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs") pod "network-metrics-daemon-l9h9s" (UID: "185f1e13-d2f1-4c57-99d8-c3eb63e20512") : secret "metrics-daemon-secret" not found Sep 09 23:03:18.161944 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.161908 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8bbb95cf7-wb54x"] Sep 09 23:03:18.163856 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.163839 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:18.167364 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.167345 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 09 23:03:18.167548 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.167530 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p5vxk\"" Sep 09 23:03:18.168478 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.168464 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 09 23:03:18.169426 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.169404 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 09 23:03:18.176589 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.176571 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 09 23:03:18.178484 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.178466 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8bbb95cf7-wb54x"] Sep 09 23:03:18.244189 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.244158 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jfnt9"] Sep 09 23:03:18.246430 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.246415 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vck7"] Sep 09 23:03:18.246543 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.246529 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jfnt9" Sep 09 23:03:18.248146 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.248131 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-87k8x"] Sep 09 23:03:18.248282 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.248264 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vck7" Sep 09 23:03:18.248851 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.248832 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 09 23:03:18.248951 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.248837 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-d8scw\"" Sep 09 23:03:18.248951 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.248837 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 09 23:03:18.249685 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.249661 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-87k8x" Sep 09 23:03:18.250634 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.250604 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-7q6zd"] Sep 09 23:03:18.250842 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.250828 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 09 23:03:18.250893 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.250830 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 09 23:03:18.252743 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.252726 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-7q6zd" Sep 09 23:03:18.255291 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.255270 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-52mwv\"" Sep 09 23:03:18.255420 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.255337 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-2lwgb\"" Sep 09 23:03:18.255491 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.255433 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 09 23:03:18.255551 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.255495 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 09 23:03:18.255850 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.255834 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6knt4\"" Sep 09 23:03:18.255939 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.255856 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 09 23:03:18.255939 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.255914 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 09 23:03:18.256186 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.256172 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 09 23:03:18.260195 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.258945 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 09 23:03:18.260195 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.258967 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-87k8x"] Sep 09 23:03:18.260195 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.259961 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jfnt9"] Sep 09 23:03:18.261276 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.261252 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vck7"] Sep 09 23:03:18.263341 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.263324 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 09 23:03:18.276960 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.276941 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-7q6zd"] Sep 09 23:03:18.280851 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.280832 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74fdfad4-a21b-45a2-9730-cfe25d4156c4-ca-trust-extracted\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:18.280949 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.280890 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74fdfad4-a21b-45a2-9730-cfe25d4156c4-installation-pull-secrets\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:18.280949 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.280920 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs9wz\" (UniqueName: \"kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-kube-api-access-bs9wz\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:18.280949 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.280943 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-bound-sa-token\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:18.281104 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.281014 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-tls\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:18.281104 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.281051 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74fdfad4-a21b-45a2-9730-cfe25d4156c4-image-registry-private-configuration\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:18.281104 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.281077 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-certificates\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:18.281295 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.281114 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74fdfad4-a21b-45a2-9730-cfe25d4156c4-trusted-ca\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:18.381912 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.381881 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35235a21-db5c-4908-971d-af808f0e4441-serving-cert\") pod \"insights-operator-76bcd6ccf5-7q6zd\" (UID: \"35235a21-db5c-4908-971d-af808f0e4441\") " pod="openshift-insights/insights-operator-76bcd6ccf5-7q6zd" Sep 09 23:03:18.382082 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.381924 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/958bade9-9642-4a24-9b2d-cd313f77c312-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9vck7\" (UID: \"958bade9-9642-4a24-9b2d-cd313f77c312\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vck7" Sep 09 23:03:18.382082 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.381956 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74fdfad4-a21b-45a2-9730-cfe25d4156c4-installation-pull-secrets\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:18.382082 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.381981 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs9wz\" (UniqueName: \"kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-kube-api-access-bs9wz\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:18.382082 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.382000 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35235a21-db5c-4908-971d-af808f0e4441-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-7q6zd\" (UID: \"35235a21-db5c-4908-971d-af808f0e4441\") " pod="openshift-insights/insights-operator-76bcd6ccf5-7q6zd" Sep 09 23:03:18.382082 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.382018 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-bound-sa-token\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:18.382082 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.382035 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5x2t\" (UniqueName: \"kubernetes.io/projected/b0439d7f-12cb-477c-965e-39e908f129ba-kube-api-access-w5x2t\") pod \"volume-data-source-validator-695c895bd6-jfnt9\" (UID: \"b0439d7f-12cb-477c-965e-39e908f129ba\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jfnt9" Sep 09 23:03:18.382403 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.382087 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2b9b\" (UniqueName: \"kubernetes.io/projected/83f5d5f1-c8ee-4704-9c16-b21ed7573b4b-kube-api-access-v2b9b\") pod \"network-check-source-685456655f-87k8x\" (UID: \"83f5d5f1-c8ee-4704-9c16-b21ed7573b4b\") " pod="openshift-network-diagnostics/network-check-source-685456655f-87k8x" Sep 09 23:03:18.382403 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.382117 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35235a21-db5c-4908-971d-af808f0e4441-tmp\") pod \"insights-operator-76bcd6ccf5-7q6zd\" (UID: \"35235a21-db5c-4908-971d-af808f0e4441\") " pod="openshift-insights/insights-operator-76bcd6ccf5-7q6zd" Sep 09 23:03:18.382403 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.382141 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/958bade9-9642-4a24-9b2d-cd313f77c312-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-9vck7\" (UID: \"958bade9-9642-4a24-9b2d-cd313f77c312\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vck7" Sep 09 23:03:18.382403 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.382202 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-tls\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:18.382403 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.382259 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74fdfad4-a21b-45a2-9730-cfe25d4156c4-image-registry-private-configuration\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:18.382403 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.382352 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-certificates\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:18.382403 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:18.382364 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 23:03:18.382403 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:18.382401 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8bbb95cf7-wb54x: secret "image-registry-tls" not found Sep 09 23:03:18.382940 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.382420 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74fdfad4-a21b-45a2-9730-cfe25d4156c4-trusted-ca\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:18.382940 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.382444 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfkfp\" (UniqueName: \"kubernetes.io/projected/958bade9-9642-4a24-9b2d-cd313f77c312-kube-api-access-mfkfp\") pod \"cluster-monitoring-operator-6cdcc96965-9vck7\" (UID: \"958bade9-9642-4a24-9b2d-cd313f77c312\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vck7" Sep 09 23:03:18.382940 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:18.382469 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-tls podName:74fdfad4-a21b-45a2-9730-cfe25d4156c4 nodeName:}" failed. No retries permitted until 2026-09-09 23:03:18.88244952 +0000 UTC m=+259.633034425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-tls") pod "image-registry-8bbb95cf7-wb54x" (UID: "74fdfad4-a21b-45a2-9730-cfe25d4156c4") : secret "image-registry-tls" not found Sep 09 23:03:18.382940 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.382507 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7jqv\" (UniqueName: \"kubernetes.io/projected/35235a21-db5c-4908-971d-af808f0e4441-kube-api-access-g7jqv\") pod \"insights-operator-76bcd6ccf5-7q6zd\" (UID: \"35235a21-db5c-4908-971d-af808f0e4441\") " pod="openshift-insights/insights-operator-76bcd6ccf5-7q6zd" Sep 09 23:03:18.382940 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.382535 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74fdfad4-a21b-45a2-9730-cfe25d4156c4-ca-trust-extracted\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:18.382940 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.382556 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/35235a21-db5c-4908-971d-af808f0e4441-snapshots\") pod \"insights-operator-76bcd6ccf5-7q6zd\" (UID: \"35235a21-db5c-4908-971d-af808f0e4441\") " pod="openshift-insights/insights-operator-76bcd6ccf5-7q6zd" Sep 09 23:03:18.382940 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.382581 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35235a21-db5c-4908-971d-af808f0e4441-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-7q6zd\" (UID: \"35235a21-db5c-4908-971d-af808f0e4441\") " pod="openshift-insights/insights-operator-76bcd6ccf5-7q6zd" Sep 09 23:03:18.383315 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.382953 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-certificates\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:18.383315 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.382953 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74fdfad4-a21b-45a2-9730-cfe25d4156c4-ca-trust-extracted\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:18.383650 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.383627 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74fdfad4-a21b-45a2-9730-cfe25d4156c4-trusted-ca\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:18.384702 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.384686 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74fdfad4-a21b-45a2-9730-cfe25d4156c4-installation-pull-secrets\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:18.384800 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.384785 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74fdfad4-a21b-45a2-9730-cfe25d4156c4-image-registry-private-configuration\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:18.399747 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.399721 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs9wz\" (UniqueName: \"kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-kube-api-access-bs9wz\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:18.400244 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.400223 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-bound-sa-token\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:18.483782 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.483699 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/958bade9-9642-4a24-9b2d-cd313f77c312-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9vck7\" (UID: \"958bade9-9642-4a24-9b2d-cd313f77c312\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vck7" Sep 09 23:03:18.483782 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.483748 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35235a21-db5c-4908-971d-af808f0e4441-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-7q6zd\" (UID: \"35235a21-db5c-4908-971d-af808f0e4441\") " pod="openshift-insights/insights-operator-76bcd6ccf5-7q6zd" Sep 09 23:03:18.483782 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.483772 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5x2t\" (UniqueName: \"kubernetes.io/projected/b0439d7f-12cb-477c-965e-39e908f129ba-kube-api-access-w5x2t\") pod \"volume-data-source-validator-695c895bd6-jfnt9\" (UID: \"b0439d7f-12cb-477c-965e-39e908f129ba\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jfnt9" Sep 09 23:03:18.483986 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.483799 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2b9b\" (UniqueName: \"kubernetes.io/projected/83f5d5f1-c8ee-4704-9c16-b21ed7573b4b-kube-api-access-v2b9b\") pod \"network-check-source-685456655f-87k8x\" (UID: \"83f5d5f1-c8ee-4704-9c16-b21ed7573b4b\") " pod="openshift-network-diagnostics/network-check-source-685456655f-87k8x" Sep 09 23:03:18.483986 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.483829 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35235a21-db5c-4908-971d-af808f0e4441-tmp\") pod \"insights-operator-76bcd6ccf5-7q6zd\" (UID: \"35235a21-db5c-4908-971d-af808f0e4441\") " pod="openshift-insights/insights-operator-76bcd6ccf5-7q6zd" Sep 09 23:03:18.483986 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:18.483832 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 23:03:18.483986 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.483855 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/958bade9-9642-4a24-9b2d-cd313f77c312-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-9vck7\" (UID: \"958bade9-9642-4a24-9b2d-cd313f77c312\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vck7" Sep 09 23:03:18.483986 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:18.483913 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/958bade9-9642-4a24-9b2d-cd313f77c312-cluster-monitoring-operator-tls podName:958bade9-9642-4a24-9b2d-cd313f77c312 nodeName:}" failed. No retries permitted until 2026-09-09 23:03:18.983891516 +0000 UTC m=+259.734476448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/958bade9-9642-4a24-9b2d-cd313f77c312-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-9vck7" (UID: "958bade9-9642-4a24-9b2d-cd313f77c312") : secret "cluster-monitoring-operator-tls" not found Sep 09 23:03:18.483986 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.483972 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfkfp\" (UniqueName: \"kubernetes.io/projected/958bade9-9642-4a24-9b2d-cd313f77c312-kube-api-access-mfkfp\") pod \"cluster-monitoring-operator-6cdcc96965-9vck7\" (UID: \"958bade9-9642-4a24-9b2d-cd313f77c312\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vck7" Sep 09 23:03:18.484284 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.483999 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7jqv\" (UniqueName: \"kubernetes.io/projected/35235a21-db5c-4908-971d-af808f0e4441-kube-api-access-g7jqv\") pod \"insights-operator-76bcd6ccf5-7q6zd\" (UID: \"35235a21-db5c-4908-971d-af808f0e4441\") " pod="openshift-insights/insights-operator-76bcd6ccf5-7q6zd" Sep 09 23:03:18.484284 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.484024 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/35235a21-db5c-4908-971d-af808f0e4441-snapshots\") pod \"insights-operator-76bcd6ccf5-7q6zd\" (UID: \"35235a21-db5c-4908-971d-af808f0e4441\") " pod="openshift-insights/insights-operator-76bcd6ccf5-7q6zd" Sep 09 23:03:18.484284 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.484046 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35235a21-db5c-4908-971d-af808f0e4441-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-7q6zd\" (UID: \"35235a21-db5c-4908-971d-af808f0e4441\") " pod="openshift-insights/insights-operator-76bcd6ccf5-7q6zd" Sep 09 23:03:18.484284 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.484068 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35235a21-db5c-4908-971d-af808f0e4441-serving-cert\") pod \"insights-operator-76bcd6ccf5-7q6zd\" (UID: \"35235a21-db5c-4908-971d-af808f0e4441\") " pod="openshift-insights/insights-operator-76bcd6ccf5-7q6zd" Sep 09 23:03:18.484284 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.484220 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35235a21-db5c-4908-971d-af808f0e4441-tmp\") pod \"insights-operator-76bcd6ccf5-7q6zd\" (UID: \"35235a21-db5c-4908-971d-af808f0e4441\") " pod="openshift-insights/insights-operator-76bcd6ccf5-7q6zd" Sep 09 23:03:18.484540 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.484336 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35235a21-db5c-4908-971d-af808f0e4441-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-7q6zd\" (UID: \"35235a21-db5c-4908-971d-af808f0e4441\") " pod="openshift-insights/insights-operator-76bcd6ccf5-7q6zd" Sep 09 23:03:18.484648 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.484597 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/958bade9-9642-4a24-9b2d-cd313f77c312-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-9vck7\" (UID: \"958bade9-9642-4a24-9b2d-cd313f77c312\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vck7" Sep 09 23:03:18.484716 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.484655 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/35235a21-db5c-4908-971d-af808f0e4441-snapshots\") pod \"insights-operator-76bcd6ccf5-7q6zd\" (UID: \"35235a21-db5c-4908-971d-af808f0e4441\") " pod="openshift-insights/insights-operator-76bcd6ccf5-7q6zd" Sep 09 23:03:18.485404 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.485365 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35235a21-db5c-4908-971d-af808f0e4441-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-7q6zd\" (UID: \"35235a21-db5c-4908-971d-af808f0e4441\") " pod="openshift-insights/insights-operator-76bcd6ccf5-7q6zd" Sep 09 23:03:18.486332 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.486317 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35235a21-db5c-4908-971d-af808f0e4441-serving-cert\") pod \"insights-operator-76bcd6ccf5-7q6zd\" (UID: \"35235a21-db5c-4908-971d-af808f0e4441\") " pod="openshift-insights/insights-operator-76bcd6ccf5-7q6zd" Sep 09 23:03:18.494870 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.494840 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5x2t\" (UniqueName: \"kubernetes.io/projected/b0439d7f-12cb-477c-965e-39e908f129ba-kube-api-access-w5x2t\") pod \"volume-data-source-validator-695c895bd6-jfnt9\" (UID: \"b0439d7f-12cb-477c-965e-39e908f129ba\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jfnt9" Sep 09 23:03:18.494983 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.494970 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfkfp\" (UniqueName: \"kubernetes.io/projected/958bade9-9642-4a24-9b2d-cd313f77c312-kube-api-access-mfkfp\") pod \"cluster-monitoring-operator-6cdcc96965-9vck7\" (UID: \"958bade9-9642-4a24-9b2d-cd313f77c312\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vck7" Sep 09 23:03:18.495179 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.495159 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7jqv\" (UniqueName: \"kubernetes.io/projected/35235a21-db5c-4908-971d-af808f0e4441-kube-api-access-g7jqv\") pod \"insights-operator-76bcd6ccf5-7q6zd\" (UID: \"35235a21-db5c-4908-971d-af808f0e4441\") " pod="openshift-insights/insights-operator-76bcd6ccf5-7q6zd" Sep 09 23:03:18.495321 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.495307 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2b9b\" (UniqueName: \"kubernetes.io/projected/83f5d5f1-c8ee-4704-9c16-b21ed7573b4b-kube-api-access-v2b9b\") pod \"network-check-source-685456655f-87k8x\" (UID: \"83f5d5f1-c8ee-4704-9c16-b21ed7573b4b\") " pod="openshift-network-diagnostics/network-check-source-685456655f-87k8x" Sep 09 23:03:18.557181 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.557157 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jfnt9" Sep 09 23:03:18.571089 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.571069 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-87k8x" Sep 09 23:03:18.575894 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.575877 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-7q6zd" Sep 09 23:03:18.827773 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.827736 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jfnt9"] Sep 09 23:03:18.830796 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:03:18.830766 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0439d7f_12cb_477c_965e_39e908f129ba.slice/crio-c542a801c463ed9dcc41d5fc4f5204b9688ce3e35c6a4d3f36555a6c2f439819 WatchSource:0}: Error finding container c542a801c463ed9dcc41d5fc4f5204b9688ce3e35c6a4d3f36555a6c2f439819: Status 404 returned error can't find the container with id c542a801c463ed9dcc41d5fc4f5204b9688ce3e35c6a4d3f36555a6c2f439819 Sep 09 23:03:18.856218 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.854397 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-87k8x"] Sep 09 23:03:18.857977 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.857588 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-7q6zd"] Sep 09 23:03:18.858613 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:03:18.858183 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f5d5f1_c8ee_4704_9c16_b21ed7573b4b.slice/crio-78bdcbac12046626f988e13a0e45a585447009802bf68bb78f3e3693be523f30 WatchSource:0}: Error finding container 78bdcbac12046626f988e13a0e45a585447009802bf68bb78f3e3693be523f30: Status 404 returned error can't find the container with id 78bdcbac12046626f988e13a0e45a585447009802bf68bb78f3e3693be523f30 Sep 09 23:03:18.863768 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:03:18.863749 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35235a21_db5c_4908_971d_af808f0e4441.slice/crio-13946d9d1da445ac5834c2d66a69ac30c569f437387c785b3e0f0e68777a0310 WatchSource:0}: Error finding container 13946d9d1da445ac5834c2d66a69ac30c569f437387c785b3e0f0e68777a0310: Status 404 returned error can't find the container with id 13946d9d1da445ac5834c2d66a69ac30c569f437387c785b3e0f0e68777a0310 Sep 09 23:03:18.887128 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.887104 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-tls\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:18.887260 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:18.887243 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 23:03:18.887308 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:18.887263 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8bbb95cf7-wb54x: secret "image-registry-tls" not found Sep 09 23:03:18.887340 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:18.887311 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-tls podName:74fdfad4-a21b-45a2-9730-cfe25d4156c4 nodeName:}" failed. No retries permitted until 2026-09-09 23:03:19.887295609 +0000 UTC m=+260.637880514 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-tls") pod "image-registry-8bbb95cf7-wb54x" (UID: "74fdfad4-a21b-45a2-9730-cfe25d4156c4") : secret "image-registry-tls" not found Sep 09 23:03:18.987966 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:18.987933 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/958bade9-9642-4a24-9b2d-cd313f77c312-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9vck7\" (UID: \"958bade9-9642-4a24-9b2d-cd313f77c312\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vck7" Sep 09 23:03:18.988138 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:18.988078 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 23:03:18.988201 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:18.988146 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/958bade9-9642-4a24-9b2d-cd313f77c312-cluster-monitoring-operator-tls podName:958bade9-9642-4a24-9b2d-cd313f77c312 nodeName:}" failed. No retries permitted until 2026-09-09 23:03:19.988126902 +0000 UTC m=+260.738711808 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/958bade9-9642-4a24-9b2d-cd313f77c312-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-9vck7" (UID: "958bade9-9642-4a24-9b2d-cd313f77c312") : secret "cluster-monitoring-operator-tls" not found Sep 09 23:03:19.601430 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:19.601368 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-87k8x" event={"ID":"83f5d5f1-c8ee-4704-9c16-b21ed7573b4b","Type":"ContainerStarted","Data":"13aa97ec4a7e56c2d5d6bb5fc1a22280777b4c377952dbc28babd3d727099476"} Sep 09 23:03:19.601906 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:19.601456 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-87k8x" event={"ID":"83f5d5f1-c8ee-4704-9c16-b21ed7573b4b","Type":"ContainerStarted","Data":"78bdcbac12046626f988e13a0e45a585447009802bf68bb78f3e3693be523f30"} Sep 09 23:03:19.603172 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:19.603141 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jfnt9" event={"ID":"b0439d7f-12cb-477c-965e-39e908f129ba","Type":"ContainerStarted","Data":"c542a801c463ed9dcc41d5fc4f5204b9688ce3e35c6a4d3f36555a6c2f439819"} Sep 09 23:03:19.604658 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:19.604626 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-7q6zd" event={"ID":"35235a21-db5c-4908-971d-af808f0e4441","Type":"ContainerStarted","Data":"13946d9d1da445ac5834c2d66a69ac30c569f437387c785b3e0f0e68777a0310"} Sep 09 23:03:19.618935 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:19.618886 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-87k8x" podStartSLOduration=1.618873687 podStartE2EDuration="1.618873687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:03:19.618305457 +0000 UTC m=+260.368890396" watchObservedRunningTime="2026-09-09 23:03:19.618873687 +0000 UTC m=+260.369458613" Sep 09 23:03:19.895440 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:19.895340 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-tls\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:19.895601 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:19.895525 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 23:03:19.895601 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:19.895548 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8bbb95cf7-wb54x: secret "image-registry-tls" not found Sep 09 23:03:19.895723 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:19.895612 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-tls podName:74fdfad4-a21b-45a2-9730-cfe25d4156c4 nodeName:}" failed. No retries permitted until 2026-09-09 23:03:21.895590798 +0000 UTC m=+262.646175710 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-tls") pod "image-registry-8bbb95cf7-wb54x" (UID: "74fdfad4-a21b-45a2-9730-cfe25d4156c4") : secret "image-registry-tls" not found Sep 09 23:03:19.996112 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:19.996076 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/958bade9-9642-4a24-9b2d-cd313f77c312-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9vck7\" (UID: \"958bade9-9642-4a24-9b2d-cd313f77c312\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vck7" Sep 09 23:03:19.996287 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:19.996232 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 23:03:19.996353 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:19.996295 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/958bade9-9642-4a24-9b2d-cd313f77c312-cluster-monitoring-operator-tls podName:958bade9-9642-4a24-9b2d-cd313f77c312 nodeName:}" failed. No retries permitted until 2026-09-09 23:03:21.99627589 +0000 UTC m=+262.746860802 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/958bade9-9642-4a24-9b2d-cd313f77c312-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-9vck7" (UID: "958bade9-9642-4a24-9b2d-cd313f77c312") : secret "cluster-monitoring-operator-tls" not found Sep 09 23:03:20.609629 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:20.609530 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jfnt9" event={"ID":"b0439d7f-12cb-477c-965e-39e908f129ba","Type":"ContainerStarted","Data":"3dd9f34eda1c73f36f4e641fc5a017e7e4d9e92d4204b32826975394ed5cdc5d"} Sep 09 23:03:20.625268 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:20.625219 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jfnt9" podStartSLOduration=1.383276771 podStartE2EDuration="2.625198277s" totalImagesPullingTime="1.241921506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:03:20.624175185 +0000 UTC m=+261.374760134" watchObservedRunningTime="2026-09-09 23:03:20.625198277 +0000 UTC m=+261.375783205" Sep 09 23:03:21.617589 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:21.617551 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-7q6zd" event={"ID":"35235a21-db5c-4908-971d-af808f0e4441","Type":"ContainerStarted","Data":"ee9d0e524852a96674d44e521d55429e7e7fc4ef8d3477059e480a2a6a0ebed3"} Sep 09 23:03:21.634714 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:21.634663 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-7q6zd" podStartSLOduration=1.5120485449999999 podStartE2EDuration="3.634649398s" totalImagesPullingTime="2.122600853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:03:21.633473554 +0000 UTC m=+262.384058497" watchObservedRunningTime="2026-09-09 23:03:21.634649398 +0000 UTC m=+262.385234360" Sep 09 23:03:21.913486 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:21.913391 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-tls\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:21.913651 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:21.913569 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 23:03:21.913651 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:21.913599 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8bbb95cf7-wb54x: secret "image-registry-tls" not found Sep 09 23:03:21.913768 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:21.913665 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-tls podName:74fdfad4-a21b-45a2-9730-cfe25d4156c4 nodeName:}" failed. No retries permitted until 2026-09-09 23:03:25.913644595 +0000 UTC m=+266.664229515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-tls") pod "image-registry-8bbb95cf7-wb54x" (UID: "74fdfad4-a21b-45a2-9730-cfe25d4156c4") : secret "image-registry-tls" not found Sep 09 23:03:21.994669 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:21.994640 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vltnr"] Sep 09 23:03:21.997167 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:21.997153 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:21.999646 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:21.999614 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 09 23:03:21.999773 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:21.999689 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nh9jh\"" Sep 09 23:03:21.999884 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:21.999871 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 09 23:03:22.010535 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:22.010514 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vltnr"] Sep 09 23:03:22.014885 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:22.014858 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/958bade9-9642-4a24-9b2d-cd313f77c312-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9vck7\" (UID: \"958bade9-9642-4a24-9b2d-cd313f77c312\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vck7" Sep 09 23:03:22.015057 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:22.015035 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 23:03:22.015159 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:22.015112 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/958bade9-9642-4a24-9b2d-cd313f77c312-cluster-monitoring-operator-tls podName:958bade9-9642-4a24-9b2d-cd313f77c312 nodeName:}" failed. No retries permitted until 2026-09-09 23:03:26.015093024 +0000 UTC m=+266.765677949 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/958bade9-9642-4a24-9b2d-cd313f77c312-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-9vck7" (UID: "958bade9-9642-4a24-9b2d-cd313f77c312") : secret "cluster-monitoring-operator-tls" not found Sep 09 23:03:22.115229 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:22.115197 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92396342-28ed-43ed-b24d-607e3855829d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:22.115229 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:22.115228 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92396342-28ed-43ed-b24d-607e3855829d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:22.115496 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:22.115281 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92396342-28ed-43ed-b24d-607e3855829d-crio-socket\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:22.115496 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:22.115337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkj4t\" (UniqueName: \"kubernetes.io/projected/92396342-28ed-43ed-b24d-607e3855829d-kube-api-access-xkj4t\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:22.115496 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:22.115400 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92396342-28ed-43ed-b24d-607e3855829d-data-volume\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:22.216123 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:22.216046 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92396342-28ed-43ed-b24d-607e3855829d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:22.216123 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:22.216078 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92396342-28ed-43ed-b24d-607e3855829d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:22.216330 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:22.216131 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92396342-28ed-43ed-b24d-607e3855829d-crio-socket\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:22.216330 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:22.216162 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 23:03:22.216330 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:22.216166 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkj4t\" (UniqueName: \"kubernetes.io/projected/92396342-28ed-43ed-b24d-607e3855829d-kube-api-access-xkj4t\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:22.216330 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:22.216236 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92396342-28ed-43ed-b24d-607e3855829d-kube-rbac-proxy-cm podName:92396342-28ed-43ed-b24d-607e3855829d nodeName:}" failed. No retries permitted until 2026-09-09 23:03:22.716217441 +0000 UTC m=+263.466802364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92396342-28ed-43ed-b24d-607e3855829d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vltnr" (UID: "92396342-28ed-43ed-b24d-607e3855829d") : configmap "kube-rbac-proxy" not found Sep 09 23:03:22.216330 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:22.216274 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:03:22.216330 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:22.216278 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92396342-28ed-43ed-b24d-607e3855829d-crio-socket\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:22.216330 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:22.216288 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92396342-28ed-43ed-b24d-607e3855829d-data-volume\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:22.216330 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:22.216330 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92396342-28ed-43ed-b24d-607e3855829d-insights-runtime-extractor-tls podName:92396342-28ed-43ed-b24d-607e3855829d nodeName:}" failed. No retries permitted until 2026-09-09 23:03:22.716312558 +0000 UTC m=+263.466897511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92396342-28ed-43ed-b24d-607e3855829d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vltnr" (UID: "92396342-28ed-43ed-b24d-607e3855829d") : secret "insights-runtime-extractor-tls" not found Sep 09 23:03:22.216636 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:22.216622 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92396342-28ed-43ed-b24d-607e3855829d-data-volume\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:22.226219 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:22.226202 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkj4t\" (UniqueName: \"kubernetes.io/projected/92396342-28ed-43ed-b24d-607e3855829d-kube-api-access-xkj4t\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:22.719696 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:22.719653 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92396342-28ed-43ed-b24d-607e3855829d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:22.720105 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:22.719730 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 23:03:22.720105 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:22.719791 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92396342-28ed-43ed-b24d-607e3855829d-kube-rbac-proxy-cm podName:92396342-28ed-43ed-b24d-607e3855829d nodeName:}" failed. No retries permitted until 2026-09-09 23:03:23.71977194 +0000 UTC m=+264.470356863 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92396342-28ed-43ed-b24d-607e3855829d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vltnr" (UID: "92396342-28ed-43ed-b24d-607e3855829d") : configmap "kube-rbac-proxy" not found Sep 09 23:03:22.720105 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:22.719788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92396342-28ed-43ed-b24d-607e3855829d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:22.720105 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:22.719903 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:03:22.720105 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:22.719946 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92396342-28ed-43ed-b24d-607e3855829d-insights-runtime-extractor-tls podName:92396342-28ed-43ed-b24d-607e3855829d nodeName:}" failed. No retries permitted until 2026-09-09 23:03:23.71993439 +0000 UTC m=+264.470519295 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92396342-28ed-43ed-b24d-607e3855829d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vltnr" (UID: "92396342-28ed-43ed-b24d-607e3855829d") : secret "insights-runtime-extractor-tls" not found Sep 09 23:03:23.725454 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:23.725419 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92396342-28ed-43ed-b24d-607e3855829d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:23.725454 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:23.725456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92396342-28ed-43ed-b24d-607e3855829d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:23.725919 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:23.725541 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 23:03:23.725919 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:23.725593 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:03:23.725919 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:23.725618 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92396342-28ed-43ed-b24d-607e3855829d-kube-rbac-proxy-cm podName:92396342-28ed-43ed-b24d-607e3855829d nodeName:}" failed. No retries permitted until 2026-09-09 23:03:25.72559886 +0000 UTC m=+266.476183785 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92396342-28ed-43ed-b24d-607e3855829d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vltnr" (UID: "92396342-28ed-43ed-b24d-607e3855829d") : configmap "kube-rbac-proxy" not found Sep 09 23:03:23.725919 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:23.725635 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92396342-28ed-43ed-b24d-607e3855829d-insights-runtime-extractor-tls podName:92396342-28ed-43ed-b24d-607e3855829d nodeName:}" failed. No retries permitted until 2026-09-09 23:03:25.725626302 +0000 UTC m=+266.476211210 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92396342-28ed-43ed-b24d-607e3855829d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vltnr" (UID: "92396342-28ed-43ed-b24d-607e3855829d") : secret "insights-runtime-extractor-tls" not found Sep 09 23:03:23.753818 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:23.753796 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6j767_6c9f70f3-174f-4bcf-9bb3-838910f4fce9/dns-node-resolver/0.log" Sep 09 23:03:24.754104 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:24.754079 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-75rcr_c864525a-b22a-4fef-ba7e-4b9541cbd717/node-ca/0.log" Sep 09 23:03:25.058915 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.058886 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xrqwx"] Sep 09 23:03:25.061801 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.061787 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xrqwx" Sep 09 23:03:25.064120 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.064099 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 09 23:03:25.065031 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.065015 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-pdcvh\"" Sep 09 23:03:25.065119 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.065032 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 09 23:03:25.069204 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.069177 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xrqwx"] Sep 09 23:03:25.137084 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.137058 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf9tj\" (UniqueName: \"kubernetes.io/projected/9a87380f-0c72-4392-9f94-95c1f3ba977f-kube-api-access-sf9tj\") pod \"migrator-fdbb8fdff-xrqwx\" (UID: \"9a87380f-0c72-4392-9f94-95c1f3ba977f\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xrqwx" Sep 09 23:03:25.238161 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.238136 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf9tj\" (UniqueName: \"kubernetes.io/projected/9a87380f-0c72-4392-9f94-95c1f3ba977f-kube-api-access-sf9tj\") pod \"migrator-fdbb8fdff-xrqwx\" (UID: \"9a87380f-0c72-4392-9f94-95c1f3ba977f\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xrqwx" Sep 09 23:03:25.245986 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.245967 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf9tj\" (UniqueName: \"kubernetes.io/projected/9a87380f-0c72-4392-9f94-95c1f3ba977f-kube-api-access-sf9tj\") pod \"migrator-fdbb8fdff-xrqwx\" (UID: \"9a87380f-0c72-4392-9f94-95c1f3ba977f\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xrqwx" Sep 09 23:03:25.371208 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.371141 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xrqwx" Sep 09 23:03:25.634885 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.634819 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xrqwx"] Sep 09 23:03:25.637844 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:03:25.637820 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a87380f_0c72_4392_9f94_95c1f3ba977f.slice/crio-166db8163f0e2ccf9a4bbbb5e138b5527f3a263ff86ffd0087fdc235df230b19 WatchSource:0}: Error finding container 166db8163f0e2ccf9a4bbbb5e138b5527f3a263ff86ffd0087fdc235df230b19: Status 404 returned error can't find the container with id 166db8163f0e2ccf9a4bbbb5e138b5527f3a263ff86ffd0087fdc235df230b19 Sep 09 23:03:25.668970 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.668942 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-nj298"] Sep 09 23:03:25.673004 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.672985 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-nj298" Sep 09 23:03:25.675424 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.675402 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jn4qp\"" Sep 09 23:03:25.675525 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.675481 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 09 23:03:25.675597 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.675580 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 09 23:03:25.679042 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.676598 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 09 23:03:25.679042 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.676856 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 09 23:03:25.682600 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.682571 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-nj298"] Sep 09 23:03:25.741026 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.741002 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0b07afaa-1a1d-4d6f-8140-18d9b9befc5f-signing-cabundle\") pod \"service-ca-75d99c484-nj298\" (UID: \"0b07afaa-1a1d-4d6f-8140-18d9b9befc5f\") " pod="openshift-service-ca/service-ca-75d99c484-nj298" Sep 09 23:03:25.741171 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.741063 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0b07afaa-1a1d-4d6f-8140-18d9b9befc5f-signing-key\") pod \"service-ca-75d99c484-nj298\" (UID: \"0b07afaa-1a1d-4d6f-8140-18d9b9befc5f\") " pod="openshift-service-ca/service-ca-75d99c484-nj298" Sep 09 23:03:25.741171 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.741090 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9f7h\" (UniqueName: \"kubernetes.io/projected/0b07afaa-1a1d-4d6f-8140-18d9b9befc5f-kube-api-access-l9f7h\") pod \"service-ca-75d99c484-nj298\" (UID: \"0b07afaa-1a1d-4d6f-8140-18d9b9befc5f\") " pod="openshift-service-ca/service-ca-75d99c484-nj298" Sep 09 23:03:25.741171 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.741152 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92396342-28ed-43ed-b24d-607e3855829d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:25.741277 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.741184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92396342-28ed-43ed-b24d-607e3855829d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:25.741277 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:25.741259 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 23:03:25.741339 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:25.741306 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:03:25.741339 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:25.741312 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92396342-28ed-43ed-b24d-607e3855829d-kube-rbac-proxy-cm podName:92396342-28ed-43ed-b24d-607e3855829d nodeName:}" failed. No retries permitted until 2026-09-09 23:03:29.741297125 +0000 UTC m=+270.491882031 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92396342-28ed-43ed-b24d-607e3855829d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vltnr" (UID: "92396342-28ed-43ed-b24d-607e3855829d") : configmap "kube-rbac-proxy" not found Sep 09 23:03:25.741444 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:25.741352 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92396342-28ed-43ed-b24d-607e3855829d-insights-runtime-extractor-tls podName:92396342-28ed-43ed-b24d-607e3855829d nodeName:}" failed. No retries permitted until 2026-09-09 23:03:29.741341228 +0000 UTC m=+270.491926133 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92396342-28ed-43ed-b24d-607e3855829d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vltnr" (UID: "92396342-28ed-43ed-b24d-607e3855829d") : secret "insights-runtime-extractor-tls" not found Sep 09 23:03:25.754841 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.754823 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9prb4_7d14c654-5bd8-4b10-aef1-32bf396708ab/serve-healthcheck-canary/0.log" Sep 09 23:03:25.841882 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.841858 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0b07afaa-1a1d-4d6f-8140-18d9b9befc5f-signing-key\") pod \"service-ca-75d99c484-nj298\" (UID: \"0b07afaa-1a1d-4d6f-8140-18d9b9befc5f\") " pod="openshift-service-ca/service-ca-75d99c484-nj298" Sep 09 23:03:25.842000 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.841893 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9f7h\" (UniqueName: \"kubernetes.io/projected/0b07afaa-1a1d-4d6f-8140-18d9b9befc5f-kube-api-access-l9f7h\") pod \"service-ca-75d99c484-nj298\" (UID: \"0b07afaa-1a1d-4d6f-8140-18d9b9befc5f\") " pod="openshift-service-ca/service-ca-75d99c484-nj298" Sep 09 23:03:25.842000 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.841954 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0b07afaa-1a1d-4d6f-8140-18d9b9befc5f-signing-cabundle\") pod \"service-ca-75d99c484-nj298\" (UID: \"0b07afaa-1a1d-4d6f-8140-18d9b9befc5f\") " pod="openshift-service-ca/service-ca-75d99c484-nj298" Sep 09 23:03:25.842609 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.842545 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0b07afaa-1a1d-4d6f-8140-18d9b9befc5f-signing-cabundle\") pod \"service-ca-75d99c484-nj298\" (UID: \"0b07afaa-1a1d-4d6f-8140-18d9b9befc5f\") " pod="openshift-service-ca/service-ca-75d99c484-nj298" Sep 09 23:03:25.844474 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.844450 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0b07afaa-1a1d-4d6f-8140-18d9b9befc5f-signing-key\") pod \"service-ca-75d99c484-nj298\" (UID: \"0b07afaa-1a1d-4d6f-8140-18d9b9befc5f\") " pod="openshift-service-ca/service-ca-75d99c484-nj298" Sep 09 23:03:25.850162 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.850136 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9f7h\" (UniqueName: \"kubernetes.io/projected/0b07afaa-1a1d-4d6f-8140-18d9b9befc5f-kube-api-access-l9f7h\") pod \"service-ca-75d99c484-nj298\" (UID: \"0b07afaa-1a1d-4d6f-8140-18d9b9befc5f\") " pod="openshift-service-ca/service-ca-75d99c484-nj298" Sep 09 23:03:25.943153 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.943087 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-tls\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:25.943251 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:25.943206 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 23:03:25.943251 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:25.943218 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8bbb95cf7-wb54x: secret "image-registry-tls" not found Sep 09 23:03:25.943322 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:25.943265 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-tls podName:74fdfad4-a21b-45a2-9730-cfe25d4156c4 nodeName:}" failed. No retries permitted until 2026-09-09 23:03:33.94325058 +0000 UTC m=+274.693835485 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-tls") pod "image-registry-8bbb95cf7-wb54x" (UID: "74fdfad4-a21b-45a2-9730-cfe25d4156c4") : secret "image-registry-tls" not found Sep 09 23:03:25.986839 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:25.986800 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-nj298" Sep 09 23:03:26.044018 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:26.043991 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/958bade9-9642-4a24-9b2d-cd313f77c312-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9vck7\" (UID: \"958bade9-9642-4a24-9b2d-cd313f77c312\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vck7" Sep 09 23:03:26.044186 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:26.044164 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 23:03:26.044259 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:26.044248 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/958bade9-9642-4a24-9b2d-cd313f77c312-cluster-monitoring-operator-tls podName:958bade9-9642-4a24-9b2d-cd313f77c312 nodeName:}" failed. No retries permitted until 2026-09-09 23:03:34.044227888 +0000 UTC m=+274.794812805 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/958bade9-9642-4a24-9b2d-cd313f77c312-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-9vck7" (UID: "958bade9-9642-4a24-9b2d-cd313f77c312") : secret "cluster-monitoring-operator-tls" not found Sep 09 23:03:26.256622 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:26.256556 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-nj298"] Sep 09 23:03:26.259288 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:03:26.259261 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b07afaa_1a1d_4d6f_8140_18d9b9befc5f.slice/crio-9b78c83efd3215a97aa17660a7f97e4716ce5e7d5780cac9b35aa7b3ac1c6497 WatchSource:0}: Error finding container 9b78c83efd3215a97aa17660a7f97e4716ce5e7d5780cac9b35aa7b3ac1c6497: Status 404 returned error can't find the container with id 9b78c83efd3215a97aa17660a7f97e4716ce5e7d5780cac9b35aa7b3ac1c6497 Sep 09 23:03:26.633070 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:26.633039 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-nj298" event={"ID":"0b07afaa-1a1d-4d6f-8140-18d9b9befc5f","Type":"ContainerStarted","Data":"9b78c83efd3215a97aa17660a7f97e4716ce5e7d5780cac9b35aa7b3ac1c6497"} Sep 09 23:03:26.634193 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:26.634170 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xrqwx" event={"ID":"9a87380f-0c72-4392-9f94-95c1f3ba977f","Type":"ContainerStarted","Data":"166db8163f0e2ccf9a4bbbb5e138b5527f3a263ff86ffd0087fdc235df230b19"} Sep 09 23:03:27.639245 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:27.639197 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xrqwx" event={"ID":"9a87380f-0c72-4392-9f94-95c1f3ba977f","Type":"ContainerStarted","Data":"c8b6efefcb0ae191b5dfde33e82a11a81670c63f082ece6d6f5bda50561983ed"} Sep 09 23:03:27.639643 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:27.639261 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xrqwx" event={"ID":"9a87380f-0c72-4392-9f94-95c1f3ba977f","Type":"ContainerStarted","Data":"1ebbdcab7147b6e8ba4aa6b9de7bcfb9dcf555884def1e834d9d98815b6e2eb7"} Sep 09 23:03:27.654883 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:27.654838 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xrqwx" podStartSLOduration=0.860981129 podStartE2EDuration="2.654823936s" totalImagesPullingTime="1.793842807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:03:27.65393575 +0000 UTC m=+268.404520693" watchObservedRunningTime="2026-09-09 23:03:27.654823936 +0000 UTC m=+268.405408856" Sep 09 23:03:28.643993 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:28.643955 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-nj298" event={"ID":"0b07afaa-1a1d-4d6f-8140-18d9b9befc5f","Type":"ContainerStarted","Data":"d42f6c5a521dd192b8d48af7b002918028c1a4763548b9785262427aa5cfca70"} Sep 09 23:03:28.662113 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:28.662065 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-nj298" podStartSLOduration=1.5412562840000001 podStartE2EDuration="3.662050393s" totalImagesPullingTime="2.120794109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:03:28.661399747 +0000 UTC m=+269.411984684" watchObservedRunningTime="2026-09-09 23:03:28.662050393 +0000 UTC m=+269.412635319" Sep 09 23:03:29.777888 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:29.777852 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92396342-28ed-43ed-b24d-607e3855829d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:29.778307 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:29.777928 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 23:03:29.778307 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:29.777990 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92396342-28ed-43ed-b24d-607e3855829d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:29.778307 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:29.778054 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92396342-28ed-43ed-b24d-607e3855829d-kube-rbac-proxy-cm podName:92396342-28ed-43ed-b24d-607e3855829d nodeName:}" failed. No retries permitted until 2026-09-09 23:03:37.778033032 +0000 UTC m=+278.528617939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92396342-28ed-43ed-b24d-607e3855829d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vltnr" (UID: "92396342-28ed-43ed-b24d-607e3855829d") : configmap "kube-rbac-proxy" not found Sep 09 23:03:29.778307 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:29.778149 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:03:29.778307 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:29.778202 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92396342-28ed-43ed-b24d-607e3855829d-insights-runtime-extractor-tls podName:92396342-28ed-43ed-b24d-607e3855829d nodeName:}" failed. No retries permitted until 2026-09-09 23:03:37.778187786 +0000 UTC m=+278.528772690 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92396342-28ed-43ed-b24d-607e3855829d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vltnr" (UID: "92396342-28ed-43ed-b24d-607e3855829d") : secret "insights-runtime-extractor-tls" not found Sep 09 23:03:34.012058 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:34.012021 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-tls\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:34.014458 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:34.014439 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-tls\") pod \"image-registry-8bbb95cf7-wb54x\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:34.073509 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:34.073468 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:34.112579 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:34.112534 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/958bade9-9642-4a24-9b2d-cd313f77c312-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9vck7\" (UID: \"958bade9-9642-4a24-9b2d-cd313f77c312\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vck7" Sep 09 23:03:34.112954 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:34.112847 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 23:03:34.113050 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:34.113031 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/958bade9-9642-4a24-9b2d-cd313f77c312-cluster-monitoring-operator-tls podName:958bade9-9642-4a24-9b2d-cd313f77c312 nodeName:}" failed. No retries permitted until 2026-09-09 23:03:50.11300566 +0000 UTC m=+290.863590579 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/958bade9-9642-4a24-9b2d-cd313f77c312-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-9vck7" (UID: "958bade9-9642-4a24-9b2d-cd313f77c312") : secret "cluster-monitoring-operator-tls" not found Sep 09 23:03:34.350263 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:34.350239 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8bbb95cf7-wb54x"] Sep 09 23:03:34.352820 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:03:34.352784 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74fdfad4_a21b_45a2_9730_cfe25d4156c4.slice/crio-bac7949097bbfd5b6fbe1e8d4c22a132d2dc7a870f940dc1ac0c9356d913e082 WatchSource:0}: Error finding container bac7949097bbfd5b6fbe1e8d4c22a132d2dc7a870f940dc1ac0c9356d913e082: Status 404 returned error can't find the container with id bac7949097bbfd5b6fbe1e8d4c22a132d2dc7a870f940dc1ac0c9356d913e082 Sep 09 23:03:34.665022 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:34.664928 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" event={"ID":"74fdfad4-a21b-45a2-9730-cfe25d4156c4","Type":"ContainerStarted","Data":"92bf4222e668ef9d53284069d54ec5cee4aad59d45e143fecb7b5f96a67b29cb"} Sep 09 23:03:34.665022 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:34.664992 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" event={"ID":"74fdfad4-a21b-45a2-9730-cfe25d4156c4","Type":"ContainerStarted","Data":"bac7949097bbfd5b6fbe1e8d4c22a132d2dc7a870f940dc1ac0c9356d913e082"} Sep 09 23:03:34.665239 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:34.665071 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:03:34.685948 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:34.685891 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" podStartSLOduration=16.685869679 podStartE2EDuration="16.685869679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:03:34.684920603 +0000 UTC m=+275.435505531" watchObservedRunningTime="2026-09-09 23:03:34.685869679 +0000 UTC m=+275.436454607" Sep 09 23:03:37.846277 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:37.846241 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92396342-28ed-43ed-b24d-607e3855829d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:37.846277 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:37.846270 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92396342-28ed-43ed-b24d-607e3855829d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:37.846957 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:37.846414 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:03:37.846957 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:37.846469 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92396342-28ed-43ed-b24d-607e3855829d-insights-runtime-extractor-tls podName:92396342-28ed-43ed-b24d-607e3855829d nodeName:}" failed. No retries permitted until 2026-09-09 23:03:53.846451142 +0000 UTC m=+294.597036051 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92396342-28ed-43ed-b24d-607e3855829d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vltnr" (UID: "92396342-28ed-43ed-b24d-607e3855829d") : secret "insights-runtime-extractor-tls" not found Sep 09 23:03:37.846957 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:37.846717 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92396342-28ed-43ed-b24d-607e3855829d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:40.351214 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:40.351174 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-wwkz5" podUID="e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec" Sep 09 23:03:40.683946 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:40.683871 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wwkz5" Sep 09 23:03:43.990557 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:43.990506 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls\") pod \"dns-default-wwkz5\" (UID: \"e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec\") " pod="openshift-dns/dns-default-wwkz5" Sep 09 23:03:43.992892 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:43.992869 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec-metrics-tls\") pod \"dns-default-wwkz5\" (UID: \"e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec\") " pod="openshift-dns/dns-default-wwkz5" Sep 09 23:03:44.287301 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:44.287277 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t7bbt\"" Sep 09 23:03:44.295685 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:44.295667 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wwkz5" Sep 09 23:03:44.567974 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:44.567896 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wwkz5"] Sep 09 23:03:44.570692 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:03:44.570667 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c19faf_3ecc_4c86_8bb6_f12f2de6cbec.slice/crio-c9df3904e86df1aea28c157cec15de9c2ded7f1c2daee63ac7f68b4db3c8a251 WatchSource:0}: Error finding container c9df3904e86df1aea28c157cec15de9c2ded7f1c2daee63ac7f68b4db3c8a251: Status 404 returned error can't find the container with id c9df3904e86df1aea28c157cec15de9c2ded7f1c2daee63ac7f68b4db3c8a251 Sep 09 23:03:44.697353 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:44.697318 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wwkz5" event={"ID":"e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec","Type":"ContainerStarted","Data":"c9df3904e86df1aea28c157cec15de9c2ded7f1c2daee63ac7f68b4db3c8a251"} Sep 09 23:03:45.200303 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.200260 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8bbb95cf7-wb54x"] Sep 09 23:03:45.242910 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.242881 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-596d7d6d66-sl4ws"] Sep 09 23:03:45.256106 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.256086 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.262167 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.262140 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-596d7d6d66-sl4ws"] Sep 09 23:03:45.300882 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.300609 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d9701aef-6851-4a0d-9381-a7625bbd0cc5-registry-certificates\") pod \"image-registry-596d7d6d66-sl4ws\" (UID: \"d9701aef-6851-4a0d-9381-a7625bbd0cc5\") " pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.300882 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.300655 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d9701aef-6851-4a0d-9381-a7625bbd0cc5-installation-pull-secrets\") pod \"image-registry-596d7d6d66-sl4ws\" (UID: \"d9701aef-6851-4a0d-9381-a7625bbd0cc5\") " pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.300882 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.300691 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d9701aef-6851-4a0d-9381-a7625bbd0cc5-ca-trust-extracted\") pod \"image-registry-596d7d6d66-sl4ws\" (UID: \"d9701aef-6851-4a0d-9381-a7625bbd0cc5\") " pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.300882 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.300766 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5l5w\" (UniqueName: \"kubernetes.io/projected/d9701aef-6851-4a0d-9381-a7625bbd0cc5-kube-api-access-s5l5w\") pod \"image-registry-596d7d6d66-sl4ws\" (UID: \"d9701aef-6851-4a0d-9381-a7625bbd0cc5\") " pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.300882 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.300821 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d9701aef-6851-4a0d-9381-a7625bbd0cc5-image-registry-private-configuration\") pod \"image-registry-596d7d6d66-sl4ws\" (UID: \"d9701aef-6851-4a0d-9381-a7625bbd0cc5\") " pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.300882 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.300851 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9701aef-6851-4a0d-9381-a7625bbd0cc5-bound-sa-token\") pod \"image-registry-596d7d6d66-sl4ws\" (UID: \"d9701aef-6851-4a0d-9381-a7625bbd0cc5\") " pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.300882 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.300884 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9701aef-6851-4a0d-9381-a7625bbd0cc5-registry-tls\") pod \"image-registry-596d7d6d66-sl4ws\" (UID: \"d9701aef-6851-4a0d-9381-a7625bbd0cc5\") " pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.301412 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.300913 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d9701aef-6851-4a0d-9381-a7625bbd0cc5-trusted-ca\") pod \"image-registry-596d7d6d66-sl4ws\" (UID: \"d9701aef-6851-4a0d-9381-a7625bbd0cc5\") " pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.401888 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.401785 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9701aef-6851-4a0d-9381-a7625bbd0cc5-registry-tls\") pod \"image-registry-596d7d6d66-sl4ws\" (UID: \"d9701aef-6851-4a0d-9381-a7625bbd0cc5\") " pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.402070 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.401907 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d9701aef-6851-4a0d-9381-a7625bbd0cc5-trusted-ca\") pod \"image-registry-596d7d6d66-sl4ws\" (UID: \"d9701aef-6851-4a0d-9381-a7625bbd0cc5\") " pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.402070 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.401994 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d9701aef-6851-4a0d-9381-a7625bbd0cc5-registry-certificates\") pod \"image-registry-596d7d6d66-sl4ws\" (UID: \"d9701aef-6851-4a0d-9381-a7625bbd0cc5\") " pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.402070 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.402021 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d9701aef-6851-4a0d-9381-a7625bbd0cc5-installation-pull-secrets\") pod \"image-registry-596d7d6d66-sl4ws\" (UID: \"d9701aef-6851-4a0d-9381-a7625bbd0cc5\") " pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.402070 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.402056 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d9701aef-6851-4a0d-9381-a7625bbd0cc5-ca-trust-extracted\") pod \"image-registry-596d7d6d66-sl4ws\" (UID: \"d9701aef-6851-4a0d-9381-a7625bbd0cc5\") " pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.402306 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.402159 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5l5w\" (UniqueName: \"kubernetes.io/projected/d9701aef-6851-4a0d-9381-a7625bbd0cc5-kube-api-access-s5l5w\") pod \"image-registry-596d7d6d66-sl4ws\" (UID: \"d9701aef-6851-4a0d-9381-a7625bbd0cc5\") " pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.402306 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.402222 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d9701aef-6851-4a0d-9381-a7625bbd0cc5-image-registry-private-configuration\") pod \"image-registry-596d7d6d66-sl4ws\" (UID: \"d9701aef-6851-4a0d-9381-a7625bbd0cc5\") " pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.402306 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.402256 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9701aef-6851-4a0d-9381-a7625bbd0cc5-bound-sa-token\") pod \"image-registry-596d7d6d66-sl4ws\" (UID: \"d9701aef-6851-4a0d-9381-a7625bbd0cc5\") " pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.402620 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.402571 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d9701aef-6851-4a0d-9381-a7625bbd0cc5-ca-trust-extracted\") pod \"image-registry-596d7d6d66-sl4ws\" (UID: \"d9701aef-6851-4a0d-9381-a7625bbd0cc5\") " pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.402904 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.402882 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d9701aef-6851-4a0d-9381-a7625bbd0cc5-registry-certificates\") pod \"image-registry-596d7d6d66-sl4ws\" (UID: \"d9701aef-6851-4a0d-9381-a7625bbd0cc5\") " pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.403345 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.403318 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d9701aef-6851-4a0d-9381-a7625bbd0cc5-trusted-ca\") pod \"image-registry-596d7d6d66-sl4ws\" (UID: \"d9701aef-6851-4a0d-9381-a7625bbd0cc5\") " pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.404847 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.404831 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d9701aef-6851-4a0d-9381-a7625bbd0cc5-installation-pull-secrets\") pod \"image-registry-596d7d6d66-sl4ws\" (UID: \"d9701aef-6851-4a0d-9381-a7625bbd0cc5\") " pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.405014 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.404987 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d9701aef-6851-4a0d-9381-a7625bbd0cc5-image-registry-private-configuration\") pod \"image-registry-596d7d6d66-sl4ws\" (UID: \"d9701aef-6851-4a0d-9381-a7625bbd0cc5\") " pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.405183 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.405145 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9701aef-6851-4a0d-9381-a7625bbd0cc5-registry-tls\") pod \"image-registry-596d7d6d66-sl4ws\" (UID: \"d9701aef-6851-4a0d-9381-a7625bbd0cc5\") " pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.411531 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.411508 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9701aef-6851-4a0d-9381-a7625bbd0cc5-bound-sa-token\") pod \"image-registry-596d7d6d66-sl4ws\" (UID: \"d9701aef-6851-4a0d-9381-a7625bbd0cc5\") " pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.411773 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.411758 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5l5w\" (UniqueName: \"kubernetes.io/projected/d9701aef-6851-4a0d-9381-a7625bbd0cc5-kube-api-access-s5l5w\") pod \"image-registry-596d7d6d66-sl4ws\" (UID: \"d9701aef-6851-4a0d-9381-a7625bbd0cc5\") " pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.567892 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.567861 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:45.876169 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:45.876097 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-596d7d6d66-sl4ws"] Sep 09 23:03:45.879301 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:03:45.879272 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9701aef_6851_4a0d_9381_a7625bbd0cc5.slice/crio-c8903fcc075d99b19b39f8e8508c3e8f4102029edbd6de92fcd5b21483e8f939 WatchSource:0}: Error finding container c8903fcc075d99b19b39f8e8508c3e8f4102029edbd6de92fcd5b21483e8f939: Status 404 returned error can't find the container with id c8903fcc075d99b19b39f8e8508c3e8f4102029edbd6de92fcd5b21483e8f939 Sep 09 23:03:46.706203 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:46.706164 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" event={"ID":"d9701aef-6851-4a0d-9381-a7625bbd0cc5","Type":"ContainerStarted","Data":"fc77b80243345d5a08d109360924463bc3fd2d5678be88b45731bc50561476b1"} Sep 09 23:03:46.706691 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:46.706252 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" event={"ID":"d9701aef-6851-4a0d-9381-a7625bbd0cc5","Type":"ContainerStarted","Data":"c8903fcc075d99b19b39f8e8508c3e8f4102029edbd6de92fcd5b21483e8f939"} Sep 09 23:03:46.706691 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:46.706319 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:03:46.725697 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:46.725644 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" podStartSLOduration=1.7256241270000001 podStartE2EDuration="1.725624127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:03:46.72408108 +0000 UTC m=+287.474666027" watchObservedRunningTime="2026-09-09 23:03:46.725624127 +0000 UTC m=+287.476209054" Sep 09 23:03:47.710878 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:47.710836 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wwkz5" event={"ID":"e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec","Type":"ContainerStarted","Data":"d3ab4af649c791ca6c82f9c1acbc6500bffc45c9953f1a955b97788733ed6e50"} Sep 09 23:03:47.711326 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:47.710912 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wwkz5" Sep 09 23:03:47.711326 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:47.710940 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wwkz5" event={"ID":"e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec","Type":"ContainerStarted","Data":"de3ecdec36a804fd976415c35a5af35f3afe5220a7ef2df97f1a843c205aa886"} Sep 09 23:03:47.726455 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:47.726392 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wwkz5" podStartSLOduration=251.43928394900001 podStartE2EDuration="4m13.726358934s" totalImagesPullingTime="2.287074985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:59:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:03:47.726306128 +0000 UTC m=+288.476891066" watchObservedRunningTime="2026-09-09 23:03:47.726358934 +0000 UTC m=+288.476943855" Sep 09 23:03:50.133266 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:50.133216 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/958bade9-9642-4a24-9b2d-cd313f77c312-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9vck7\" (UID: \"958bade9-9642-4a24-9b2d-cd313f77c312\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vck7" Sep 09 23:03:50.135840 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:50.135817 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/958bade9-9642-4a24-9b2d-cd313f77c312-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-9vck7\" (UID: \"958bade9-9642-4a24-9b2d-cd313f77c312\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vck7" Sep 09 23:03:50.365351 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:50.365318 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vck7" Sep 09 23:03:50.639726 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:50.637569 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vck7"] Sep 09 23:03:50.642859 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:03:50.642828 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod958bade9_9642_4a24_9b2d_cd313f77c312.slice/crio-a27beb5b51db1ea7106c30edee7521cca700b1bda7d7b2df9b7e63f4dde774f3 WatchSource:0}: Error finding container a27beb5b51db1ea7106c30edee7521cca700b1bda7d7b2df9b7e63f4dde774f3: Status 404 returned error can't find the container with id a27beb5b51db1ea7106c30edee7521cca700b1bda7d7b2df9b7e63f4dde774f3 Sep 09 23:03:50.721442 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:50.721350 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vck7" event={"ID":"958bade9-9642-4a24-9b2d-cd313f77c312","Type":"ContainerStarted","Data":"a27beb5b51db1ea7106c30edee7521cca700b1bda7d7b2df9b7e63f4dde774f3"} Sep 09 23:03:53.731655 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:53.731623 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vck7" event={"ID":"958bade9-9642-4a24-9b2d-cd313f77c312","Type":"ContainerStarted","Data":"0dfe816b7461aa69dc63ab675459059bc6d8985f3f4b8e46398022fd864d4082"} Sep 09 23:03:53.747495 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:53.747447 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-9vck7" podStartSLOduration=33.561326314 podStartE2EDuration="35.747430121s" totalImagesPullingTime="2.186103807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:03:53.746922126 +0000 UTC m=+294.497507072" watchObservedRunningTime="2026-09-09 23:03:53.747430121 +0000 UTC m=+294.498015054" Sep 09 23:03:53.846742 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:03:53.846699 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-l9h9s" podUID="185f1e13-d2f1-4c57-99d8-c3eb63e20512" Sep 09 23:03:53.865414 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:53.865366 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92396342-28ed-43ed-b24d-607e3855829d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:53.867728 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:53.867707 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92396342-28ed-43ed-b24d-607e3855829d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vltnr\" (UID: \"92396342-28ed-43ed-b24d-607e3855829d\") " pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:54.107447 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:54.107403 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vltnr" Sep 09 23:03:54.377344 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:54.377319 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vltnr"] Sep 09 23:03:54.380040 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:03:54.380014 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92396342_28ed_43ed_b24d_607e3855829d.slice/crio-a941157195adcfb088f7baf7c625a0f3de6d5623b30781ee9439b4fa44c1cf08 WatchSource:0}: Error finding container a941157195adcfb088f7baf7c625a0f3de6d5623b30781ee9439b4fa44c1cf08: Status 404 returned error can't find the container with id a941157195adcfb088f7baf7c625a0f3de6d5623b30781ee9439b4fa44c1cf08 Sep 09 23:03:54.736644 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:54.736564 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vltnr" event={"ID":"92396342-28ed-43ed-b24d-607e3855829d","Type":"ContainerStarted","Data":"8de3ca2cd5a39303360bc1694f54c17dd46581ada6fdbda969fc09fc63aa3e89"} Sep 09 23:03:54.736644 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:54.736621 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vltnr" event={"ID":"92396342-28ed-43ed-b24d-607e3855829d","Type":"ContainerStarted","Data":"a941157195adcfb088f7baf7c625a0f3de6d5623b30781ee9439b4fa44c1cf08"} Sep 09 23:03:55.206030 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:55.206001 2578 patch_prober.go:28] interesting pod/image-registry-8bbb95cf7-wb54x container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 09 23:03:55.206162 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:55.206048 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" podUID="74fdfad4-a21b-45a2-9730-cfe25d4156c4" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 09 23:03:55.742089 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:55.742059 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vltnr" event={"ID":"92396342-28ed-43ed-b24d-607e3855829d","Type":"ContainerStarted","Data":"f8c5864f4265711d0c9419909ca9e1cc94e7802d6aba47135d99275a42f77f22"} Sep 09 23:03:56.747130 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:56.747081 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vltnr" event={"ID":"92396342-28ed-43ed-b24d-607e3855829d","Type":"ContainerStarted","Data":"1413d4d7c252633f265144cd17674c3adfeb364967d3c50c3032dca9d33bdb83"} Sep 09 23:03:56.766258 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:56.766211 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vltnr" podStartSLOduration=33.688384549 podStartE2EDuration="35.766195987s" totalImagesPullingTime="2.077811438s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:03:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:03:56.764832204 +0000 UTC m=+297.515417142" watchObservedRunningTime="2026-09-09 23:03:56.766195987 +0000 UTC m=+297.516780945" Sep 09 23:03:57.716248 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:57.716220 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wwkz5" Sep 09 23:03:59.727963 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.727934 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s8wp2"] Sep 09 23:03:59.735017 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.734978 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.737557 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.737527 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 09 23:03:59.738076 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.738050 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 09 23:03:59.738523 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.738504 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jc9mn\"" Sep 09 23:03:59.738694 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.738673 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 09 23:03:59.738774 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.738529 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 09 23:03:59.756689 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.756671 2578 kubelet.go:1648] "Image garbage collection succeeded" Sep 09 23:03:59.809557 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.809424 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60dc9d97-70e5-4683-85c0-539aa019f6ac-sys\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.809557 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.809472 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/60dc9d97-70e5-4683-85c0-539aa019f6ac-node-exporter-tls\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.809557 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.809489 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60dc9d97-70e5-4683-85c0-539aa019f6ac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.809557 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.809531 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxv95\" (UniqueName: \"kubernetes.io/projected/60dc9d97-70e5-4683-85c0-539aa019f6ac-kube-api-access-mxv95\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.809557 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.809552 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60dc9d97-70e5-4683-85c0-539aa019f6ac-metrics-client-ca\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.821729 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.809583 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/60dc9d97-70e5-4683-85c0-539aa019f6ac-node-exporter-textfile\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.821729 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.809616 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/60dc9d97-70e5-4683-85c0-539aa019f6ac-node-exporter-accelerators-collector-config\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.821729 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.809635 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/60dc9d97-70e5-4683-85c0-539aa019f6ac-root\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.821729 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.809665 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/60dc9d97-70e5-4683-85c0-539aa019f6ac-node-exporter-wtmp\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.910604 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.910576 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/60dc9d97-70e5-4683-85c0-539aa019f6ac-node-exporter-wtmp\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.910791 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.910620 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60dc9d97-70e5-4683-85c0-539aa019f6ac-sys\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.910791 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.910679 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60dc9d97-70e5-4683-85c0-539aa019f6ac-sys\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.910791 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.910700 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/60dc9d97-70e5-4683-85c0-539aa019f6ac-node-exporter-tls\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.910791 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.910715 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60dc9d97-70e5-4683-85c0-539aa019f6ac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.910791 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.910770 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/60dc9d97-70e5-4683-85c0-539aa019f6ac-node-exporter-wtmp\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.911062 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.910823 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxv95\" (UniqueName: \"kubernetes.io/projected/60dc9d97-70e5-4683-85c0-539aa019f6ac-kube-api-access-mxv95\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.911062 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.910858 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60dc9d97-70e5-4683-85c0-539aa019f6ac-metrics-client-ca\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.911062 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.910898 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/60dc9d97-70e5-4683-85c0-539aa019f6ac-node-exporter-textfile\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.911062 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.910923 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/60dc9d97-70e5-4683-85c0-539aa019f6ac-node-exporter-accelerators-collector-config\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.911062 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.910950 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/60dc9d97-70e5-4683-85c0-539aa019f6ac-root\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.911306 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.911202 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/60dc9d97-70e5-4683-85c0-539aa019f6ac-node-exporter-textfile\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.911306 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.911265 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/60dc9d97-70e5-4683-85c0-539aa019f6ac-root\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.911608 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.911590 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60dc9d97-70e5-4683-85c0-539aa019f6ac-metrics-client-ca\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.911892 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.911869 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/60dc9d97-70e5-4683-85c0-539aa019f6ac-node-exporter-accelerators-collector-config\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.913192 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.913174 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60dc9d97-70e5-4683-85c0-539aa019f6ac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.913443 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.913428 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/60dc9d97-70e5-4683-85c0-539aa019f6ac-node-exporter-tls\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:03:59.918922 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:03:59.918899 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxv95\" (UniqueName: \"kubernetes.io/projected/60dc9d97-70e5-4683-85c0-539aa019f6ac-kube-api-access-mxv95\") pod \"node-exporter-s8wp2\" (UID: \"60dc9d97-70e5-4683-85c0-539aa019f6ac\") " pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:04:00.047786 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.047750 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s8wp2" Sep 09 23:04:00.059256 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:04:00.059225 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60dc9d97_70e5_4683_85c0_539aa019f6ac.slice/crio-201b2ce4fe01cc1a7e9d3955c71e4b4fdaad2a47e2a7d9e919f035aadcb6a16f WatchSource:0}: Error finding container 201b2ce4fe01cc1a7e9d3955c71e4b4fdaad2a47e2a7d9e919f035aadcb6a16f: Status 404 returned error can't find the container with id 201b2ce4fe01cc1a7e9d3955c71e4b4fdaad2a47e2a7d9e919f035aadcb6a16f Sep 09 23:04:00.764159 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.764116 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s8wp2" event={"ID":"60dc9d97-70e5-4683-85c0-539aa019f6ac","Type":"ContainerStarted","Data":"201b2ce4fe01cc1a7e9d3955c71e4b4fdaad2a47e2a7d9e919f035aadcb6a16f"} Sep 09 23:04:00.781906 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.781875 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 23:04:00.785804 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.785767 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.788147 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.788109 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 09 23:04:00.788275 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.788238 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 09 23:04:00.789536 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.788582 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 09 23:04:00.789536 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.788773 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4v4r9\"" Sep 09 23:04:00.789536 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.788977 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 09 23:04:00.789536 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.789139 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 09 23:04:00.789536 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.789305 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 09 23:04:00.789536 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.789358 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 09 23:04:00.789896 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.789601 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 09 23:04:00.789896 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.789762 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 09 23:04:00.801058 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.801036 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 23:04:00.819162 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.819128 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.819315 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.819173 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtvqp\" (UniqueName: \"kubernetes.io/projected/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-kube-api-access-jtvqp\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.819315 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.819199 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-config-out\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.819315 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.819267 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-web-config\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.819315 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.819307 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.819498 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.819331 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.819498 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.819347 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.819498 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.819442 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.819498 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.819469 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.819498 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.819490 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.819668 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.819509 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-config-volume\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.819668 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.819600 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.819668 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.819636 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.920634 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.920598 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.920634 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.920649 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.920889 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.920681 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.920889 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.920710 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-config-volume\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.920889 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.920749 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.920889 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.920771 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.920889 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:04:00.920805 2578 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 09 23:04:00.920889 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.920820 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.920889 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.920840 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtvqp\" (UniqueName: \"kubernetes.io/projected/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-kube-api-access-jtvqp\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.920889 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:04:00.920879 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-main-tls podName:e8ae840a-52ae-4e68-8b87-a5b48b9aab6f nodeName:}" failed. No retries permitted until 2026-09-09 23:04:01.420858488 +0000 UTC m=+302.171443410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f") : secret "alertmanager-main-tls" not found Sep 09 23:04:00.921300 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.920903 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-config-out\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.921300 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.920944 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-web-config\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.921300 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.920977 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.921300 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.921009 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.921300 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.921036 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.925245 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.924507 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.925245 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.925176 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.925473 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.925449 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.925582 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:04:00.925565 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-alertmanager-trusted-ca-bundle podName:e8ae840a-52ae-4e68-8b87-a5b48b9aab6f nodeName:}" failed. No retries permitted until 2026-09-09 23:04:01.425547918 +0000 UTC m=+302.176132840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f") : configmap references non-existent config key: ca-bundle.crt Sep 09 23:04:00.926423 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.926002 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.926423 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.926086 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.926423 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.926346 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-web-config\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.927257 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.926931 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-config-out\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.927257 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.927187 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.927881 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.927854 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-config-volume\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.928082 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.928045 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:00.930631 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:00.930537 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtvqp\" (UniqueName: \"kubernetes.io/projected/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-kube-api-access-jtvqp\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:01.425721 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:01.425633 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:01.425721 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:01.425700 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:01.426436 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:01.426411 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:01.428250 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:01.428220 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:01.697183 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:01.697100 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:04:01.772218 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:01.772184 2578 generic.go:356] "Generic (PLEG): container finished" podID="60dc9d97-70e5-4683-85c0-539aa019f6ac" containerID="cef9b15cf3701cca2bfbff9f8a5ee1d3a799fe08e6107b43318368d760933cf7" exitCode=0 Sep 09 23:04:01.772673 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:01.772262 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s8wp2" event={"ID":"60dc9d97-70e5-4683-85c0-539aa019f6ac","Type":"ContainerDied","Data":"cef9b15cf3701cca2bfbff9f8a5ee1d3a799fe08e6107b43318368d760933cf7"} Sep 09 23:04:01.975954 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:01.975871 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 23:04:01.978743 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:04:01.978710 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ae840a_52ae_4e68_8b87_a5b48b9aab6f.slice/crio-c85041c6764d48e6b1e58985fd3de86da1578fc090adbad1f58a6bb4db6b42d3 WatchSource:0}: Error finding container c85041c6764d48e6b1e58985fd3de86da1578fc090adbad1f58a6bb4db6b42d3: Status 404 returned error can't find the container with id c85041c6764d48e6b1e58985fd3de86da1578fc090adbad1f58a6bb4db6b42d3 Sep 09 23:04:01.980842 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:01.980826 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 23:04:02.780301 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:02.780263 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s8wp2" event={"ID":"60dc9d97-70e5-4683-85c0-539aa019f6ac","Type":"ContainerStarted","Data":"b3a5cfffddc10ac0368ee33242eb0f84e019545459c3a6fb4d3f3f2cb11edd1d"} Sep 09 23:04:02.780818 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:02.780329 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s8wp2" event={"ID":"60dc9d97-70e5-4683-85c0-539aa019f6ac","Type":"ContainerStarted","Data":"0521c09f8b91adf9cd3f195edf741f9e5eec1a6a2a9f7c41b269bb3ade89deae"} Sep 09 23:04:02.782178 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:02.782147 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f","Type":"ContainerStarted","Data":"c85041c6764d48e6b1e58985fd3de86da1578fc090adbad1f58a6bb4db6b42d3"} Sep 09 23:04:02.801810 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:02.801753 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s8wp2" podStartSLOduration=2.705428909 podStartE2EDuration="3.801731078s" totalImagesPullingTime="1.027625207s" totalInitContainerRuntime="68.676962ms" isStatefulPod=true podCreationTimestamp="2026-09-09 23:03:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:04:02.800175529 +0000 UTC m=+303.550760474" watchObservedRunningTime="2026-09-09 23:04:02.801731078 +0000 UTC m=+303.552316006" Sep 09 23:04:03.791009 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:03.790976 2578 generic.go:356] "Generic (PLEG): container finished" podID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerID="611f1147ca45486c633f1d4e47137a46ad01d1027cd6b9d08eaedeb044ed917e" exitCode=0 Sep 09 23:04:03.791397 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:03.791069 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f","Type":"ContainerDied","Data":"611f1147ca45486c633f1d4e47137a46ad01d1027cd6b9d08eaedeb044ed917e"} Sep 09 23:04:04.536699 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:04.536663 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-8g6z5"] Sep 09 23:04:04.539732 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:04.539717 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8g6z5" Sep 09 23:04:04.543360 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:04.543338 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-rjmjl\"" Sep 09 23:04:04.543506 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:04.543480 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 09 23:04:04.559163 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:04.559130 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-8g6z5"] Sep 09 23:04:04.656978 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:04.656944 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6330f214-729e-404e-9b73-c907fe394743-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-8g6z5\" (UID: \"6330f214-729e-404e-9b73-c907fe394743\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8g6z5" Sep 09 23:04:04.657159 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:04.656998 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhs5d\" (UniqueName: \"kubernetes.io/projected/6330f214-729e-404e-9b73-c907fe394743-kube-api-access-zhs5d\") pod \"monitoring-plugin-85bbbbbf4-8g6z5\" (UID: \"6330f214-729e-404e-9b73-c907fe394743\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8g6z5" Sep 09 23:04:04.757496 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:04.757466 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhs5d\" (UniqueName: \"kubernetes.io/projected/6330f214-729e-404e-9b73-c907fe394743-kube-api-access-zhs5d\") pod \"monitoring-plugin-85bbbbbf4-8g6z5\" (UID: \"6330f214-729e-404e-9b73-c907fe394743\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8g6z5" Sep 09 23:04:04.757698 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:04.757573 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6330f214-729e-404e-9b73-c907fe394743-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-8g6z5\" (UID: \"6330f214-729e-404e-9b73-c907fe394743\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8g6z5" Sep 09 23:04:04.760020 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:04.759998 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6330f214-729e-404e-9b73-c907fe394743-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-8g6z5\" (UID: \"6330f214-729e-404e-9b73-c907fe394743\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8g6z5" Sep 09 23:04:04.773417 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:04.773341 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhs5d\" (UniqueName: \"kubernetes.io/projected/6330f214-729e-404e-9b73-c907fe394743-kube-api-access-zhs5d\") pod \"monitoring-plugin-85bbbbbf4-8g6z5\" (UID: \"6330f214-729e-404e-9b73-c907fe394743\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8g6z5" Sep 09 23:04:04.848706 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:04.848623 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8g6z5" Sep 09 23:04:05.128237 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:05.128141 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-8g6z5"] Sep 09 23:04:05.206985 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:05.206957 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:04:05.272595 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:04:05.272563 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6330f214_729e_404e_9b73_c907fe394743.slice/crio-72f614c632c9f1f2786b29d4464ee7908863ed8cf69dc94dc3dda5226d039bcb WatchSource:0}: Error finding container 72f614c632c9f1f2786b29d4464ee7908863ed8cf69dc94dc3dda5226d039bcb: Status 404 returned error can't find the container with id 72f614c632c9f1f2786b29d4464ee7908863ed8cf69dc94dc3dda5226d039bcb Sep 09 23:04:05.802502 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:05.802465 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f","Type":"ContainerStarted","Data":"8c2a8a87a34f4637d97c2950a66389b53a59205967ce35135d396c3f0359309c"} Sep 09 23:04:05.802680 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:05.802543 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f","Type":"ContainerStarted","Data":"cec9bc1e520b652033ac8a90132cc465a399e7f3a08bf341e7b522394ee15b34"} Sep 09 23:04:05.802680 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:05.802580 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f","Type":"ContainerStarted","Data":"3a42ebbcead5ddc022fe1e76d17484702625ea59ee53400e65c560a0ccc6d5a7"} Sep 09 23:04:05.802680 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:05.802615 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f","Type":"ContainerStarted","Data":"ffe965fdf35fa4d4650bb88d7666120cef8b70f1b1ab7c6727880ad066cc9236"} Sep 09 23:04:05.802680 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:05.802648 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f","Type":"ContainerStarted","Data":"ffd3c1864c6ee5bdf68f3619b4814fc1c173ce4febe3551c1f2386af2d17e9a3"} Sep 09 23:04:05.803617 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:05.803594 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8g6z5" event={"ID":"6330f214-729e-404e-9b73-c907fe394743","Type":"ContainerStarted","Data":"72f614c632c9f1f2786b29d4464ee7908863ed8cf69dc94dc3dda5226d039bcb"} Sep 09 23:04:05.846097 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:05.846069 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 23:04:06.811967 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:06.811899 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f","Type":"ContainerStarted","Data":"733094299f8dfd71c1670b69f698cbdf4bee17bf24bc74de3b5b25f922de8d09"} Sep 09 23:04:06.862454 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:06.862408 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.1312206639999998 podStartE2EDuration="6.86237419s" totalImagesPullingTime="3.713082181s" totalInitContainerRuntime="18.071345ms" isStatefulPod=false podCreationTimestamp="2026-09-09 23:04:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:04:06.858697358 +0000 UTC m=+307.609282319" watchObservedRunningTime="2026-09-09 23:04:06.86237419 +0000 UTC m=+307.612959116" Sep 09 23:04:07.715985 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:07.715894 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-596d7d6d66-sl4ws" Sep 09 23:04:07.817040 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:07.817009 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8g6z5" event={"ID":"6330f214-729e-404e-9b73-c907fe394743","Type":"ContainerStarted","Data":"4965a70de7adc4df27db842f875332d8382436509676b9dec18c187df265e097"} Sep 09 23:04:07.817508 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:07.817486 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8g6z5" Sep 09 23:04:07.821852 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:07.821831 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8g6z5" Sep 09 23:04:07.847085 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:07.847042 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8g6z5" podStartSLOduration=1.837420506 podStartE2EDuration="3.847028224s" totalImagesPullingTime="2.009607718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:04:07.845996482 +0000 UTC m=+308.596581420" watchObservedRunningTime="2026-09-09 23:04:07.847028224 +0000 UTC m=+308.597613151" Sep 09 23:04:10.225773 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.225738 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" podUID="74fdfad4-a21b-45a2-9730-cfe25d4156c4" containerName="registry" containerID="cri-o://92bf4222e668ef9d53284069d54ec5cee4aad59d45e143fecb7b5f96a67b29cb" gracePeriod=30 Sep 09 23:04:10.461480 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.461455 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:04:10.608873 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.608839 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-tls\") pod \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " Sep 09 23:04:10.608873 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.608877 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-bound-sa-token\") pod \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " Sep 09 23:04:10.609098 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.608904 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74fdfad4-a21b-45a2-9730-cfe25d4156c4-ca-trust-extracted\") pod \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " Sep 09 23:04:10.609098 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.608955 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74fdfad4-a21b-45a2-9730-cfe25d4156c4-image-registry-private-configuration\") pod \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " Sep 09 23:04:10.609098 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.608976 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-certificates\") pod \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " Sep 09 23:04:10.609098 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.609018 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bs9wz\" (UniqueName: \"kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-kube-api-access-bs9wz\") pod \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " Sep 09 23:04:10.609098 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.609057 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74fdfad4-a21b-45a2-9730-cfe25d4156c4-trusted-ca\") pod \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " Sep 09 23:04:10.609098 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.609095 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74fdfad4-a21b-45a2-9730-cfe25d4156c4-installation-pull-secrets\") pod \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\" (UID: \"74fdfad4-a21b-45a2-9730-cfe25d4156c4\") " Sep 09 23:04:10.609537 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.609481 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "74fdfad4-a21b-45a2-9730-cfe25d4156c4" (UID: "74fdfad4-a21b-45a2-9730-cfe25d4156c4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:04:10.610016 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.609975 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74fdfad4-a21b-45a2-9730-cfe25d4156c4-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "74fdfad4-a21b-45a2-9730-cfe25d4156c4" (UID: "74fdfad4-a21b-45a2-9730-cfe25d4156c4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:04:10.611642 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.611616 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74fdfad4-a21b-45a2-9730-cfe25d4156c4-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "74fdfad4-a21b-45a2-9730-cfe25d4156c4" (UID: "74fdfad4-a21b-45a2-9730-cfe25d4156c4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:04:10.611802 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.611782 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-kube-api-access-bs9wz" (OuterVolumeSpecName: "kube-api-access-bs9wz") pod "74fdfad4-a21b-45a2-9730-cfe25d4156c4" (UID: "74fdfad4-a21b-45a2-9730-cfe25d4156c4"). InnerVolumeSpecName "kube-api-access-bs9wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:04:10.611871 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.611801 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "74fdfad4-a21b-45a2-9730-cfe25d4156c4" (UID: "74fdfad4-a21b-45a2-9730-cfe25d4156c4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:04:10.611871 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.611803 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74fdfad4-a21b-45a2-9730-cfe25d4156c4-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "74fdfad4-a21b-45a2-9730-cfe25d4156c4" (UID: "74fdfad4-a21b-45a2-9730-cfe25d4156c4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:04:10.611871 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.611847 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "74fdfad4-a21b-45a2-9730-cfe25d4156c4" (UID: "74fdfad4-a21b-45a2-9730-cfe25d4156c4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:04:10.617880 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.617857 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fdfad4-a21b-45a2-9730-cfe25d4156c4-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "74fdfad4-a21b-45a2-9730-cfe25d4156c4" (UID: "74fdfad4-a21b-45a2-9730-cfe25d4156c4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:04:10.709941 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.709908 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs9wz\" (UniqueName: \"kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-kube-api-access-bs9wz\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:04:10.709941 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.709934 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74fdfad4-a21b-45a2-9730-cfe25d4156c4-trusted-ca\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:04:10.709941 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.709944 2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74fdfad4-a21b-45a2-9730-cfe25d4156c4-installation-pull-secrets\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:04:10.710151 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.709953 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-tls\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:04:10.710151 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.709961 2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74fdfad4-a21b-45a2-9730-cfe25d4156c4-bound-sa-token\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:04:10.710151 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.709968 2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74fdfad4-a21b-45a2-9730-cfe25d4156c4-ca-trust-extracted\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:04:10.710151 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.709977 2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74fdfad4-a21b-45a2-9730-cfe25d4156c4-image-registry-private-configuration\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:04:10.710151 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.709985 2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74fdfad4-a21b-45a2-9730-cfe25d4156c4-registry-certificates\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:04:10.829036 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.828993 2578 generic.go:356] "Generic (PLEG): container finished" podID="74fdfad4-a21b-45a2-9730-cfe25d4156c4" containerID="92bf4222e668ef9d53284069d54ec5cee4aad59d45e143fecb7b5f96a67b29cb" exitCode=0 Sep 09 23:04:10.829216 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.829062 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" Sep 09 23:04:10.829216 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.829078 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" event={"ID":"74fdfad4-a21b-45a2-9730-cfe25d4156c4","Type":"ContainerDied","Data":"92bf4222e668ef9d53284069d54ec5cee4aad59d45e143fecb7b5f96a67b29cb"} Sep 09 23:04:10.829216 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.829150 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8bbb95cf7-wb54x" event={"ID":"74fdfad4-a21b-45a2-9730-cfe25d4156c4","Type":"ContainerDied","Data":"bac7949097bbfd5b6fbe1e8d4c22a132d2dc7a870f940dc1ac0c9356d913e082"} Sep 09 23:04:10.829216 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.829171 2578 scope.go:117] "RemoveContainer" containerID="92bf4222e668ef9d53284069d54ec5cee4aad59d45e143fecb7b5f96a67b29cb" Sep 09 23:04:10.842171 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.842156 2578 scope.go:117] "RemoveContainer" containerID="92bf4222e668ef9d53284069d54ec5cee4aad59d45e143fecb7b5f96a67b29cb" Sep 09 23:04:10.842506 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:04:10.842480 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92bf4222e668ef9d53284069d54ec5cee4aad59d45e143fecb7b5f96a67b29cb\": container with ID starting with 92bf4222e668ef9d53284069d54ec5cee4aad59d45e143fecb7b5f96a67b29cb not found: ID does not exist" containerID="92bf4222e668ef9d53284069d54ec5cee4aad59d45e143fecb7b5f96a67b29cb" Sep 09 23:04:10.842568 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.842518 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92bf4222e668ef9d53284069d54ec5cee4aad59d45e143fecb7b5f96a67b29cb"} err="failed to get container status \"92bf4222e668ef9d53284069d54ec5cee4aad59d45e143fecb7b5f96a67b29cb\": rpc error: code = NotFound desc = could not find container \"92bf4222e668ef9d53284069d54ec5cee4aad59d45e143fecb7b5f96a67b29cb\": container with ID starting with 92bf4222e668ef9d53284069d54ec5cee4aad59d45e143fecb7b5f96a67b29cb not found: ID does not exist" Sep 09 23:04:10.854235 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.854214 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8bbb95cf7-wb54x"] Sep 09 23:04:10.858141 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:10.858119 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8bbb95cf7-wb54x"] Sep 09 23:04:11.849854 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:11.849821 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74fdfad4-a21b-45a2-9730-cfe25d4156c4" path="/var/lib/kubelet/pods/74fdfad4-a21b-45a2-9730-cfe25d4156c4/volumes" Sep 09 23:04:31.902557 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:31.902473 2578 generic.go:356] "Generic (PLEG): container finished" podID="35235a21-db5c-4908-971d-af808f0e4441" containerID="ee9d0e524852a96674d44e521d55429e7e7fc4ef8d3477059e480a2a6a0ebed3" exitCode=0 Sep 09 23:04:31.902557 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:31.902546 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-7q6zd" event={"ID":"35235a21-db5c-4908-971d-af808f0e4441","Type":"ContainerDied","Data":"ee9d0e524852a96674d44e521d55429e7e7fc4ef8d3477059e480a2a6a0ebed3"} Sep 09 23:04:31.902938 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:31.902849 2578 scope.go:117] "RemoveContainer" containerID="ee9d0e524852a96674d44e521d55429e7e7fc4ef8d3477059e480a2a6a0ebed3" Sep 09 23:04:32.909314 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:04:32.909275 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-7q6zd" event={"ID":"35235a21-db5c-4908-971d-af808f0e4441","Type":"ContainerStarted","Data":"58233fe55d2033035827b2f730888c413d8520ffe5138d2710ffcf4ca4b918c8"} Sep 09 23:05:13.710822 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:13.710782 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs\") pod \"network-metrics-daemon-l9h9s\" (UID: \"185f1e13-d2f1-4c57-99d8-c3eb63e20512\") " pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 23:05:13.713167 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:13.713146 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185f1e13-d2f1-4c57-99d8-c3eb63e20512-metrics-certs\") pod \"network-metrics-daemon-l9h9s\" (UID: \"185f1e13-d2f1-4c57-99d8-c3eb63e20512\") " pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 23:05:13.949858 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:13.949827 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vjfnz\"" Sep 09 23:05:13.957864 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:13.957840 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9h9s" Sep 09 23:05:14.224930 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:14.224862 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l9h9s"] Sep 09 23:05:14.227459 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:05:14.227433 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185f1e13_d2f1_4c57_99d8_c3eb63e20512.slice/crio-da22c8bf4caa6fb2dd4b236d0298302ab702642df887e8a3025e3f2092e2f778 WatchSource:0}: Error finding container da22c8bf4caa6fb2dd4b236d0298302ab702642df887e8a3025e3f2092e2f778: Status 404 returned error can't find the container with id da22c8bf4caa6fb2dd4b236d0298302ab702642df887e8a3025e3f2092e2f778 Sep 09 23:05:15.064397 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:15.064343 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l9h9s" event={"ID":"185f1e13-d2f1-4c57-99d8-c3eb63e20512","Type":"ContainerStarted","Data":"da22c8bf4caa6fb2dd4b236d0298302ab702642df887e8a3025e3f2092e2f778"} Sep 09 23:05:16.069823 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:16.069788 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l9h9s" event={"ID":"185f1e13-d2f1-4c57-99d8-c3eb63e20512","Type":"ContainerStarted","Data":"cce895282b284c0f40077fc49166419cf8e6678a0de2f6a67745362e20686b64"} Sep 09 23:05:16.070293 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:16.069849 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l9h9s" event={"ID":"185f1e13-d2f1-4c57-99d8-c3eb63e20512","Type":"ContainerStarted","Data":"1f7fa8ebc6c2860a1dc80a55c3930d88b84e2aec9e9034475e0b25a0f67c358c"} Sep 09 23:05:16.086954 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:16.086907 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-l9h9s" podStartSLOduration=375.111097043 podStartE2EDuration="6m16.086891624s" totalImagesPullingTime="975.794581ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 22:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:05:16.085411978 +0000 UTC m=+376.835996922" watchObservedRunningTime="2026-09-09 23:05:16.086891624 +0000 UTC m=+376.837476551" Sep 09 23:05:20.028743 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:20.028705 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 23:05:20.029204 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:20.028992 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="alertmanager" containerID="cri-o://ffd3c1864c6ee5bdf68f3619b4814fc1c173ce4febe3551c1f2386af2d17e9a3" gracePeriod=120 Sep 09 23:05:20.029204 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:20.029029 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="kube-rbac-proxy" containerID="cri-o://cec9bc1e520b652033ac8a90132cc465a399e7f3a08bf341e7b522394ee15b34" gracePeriod=120 Sep 09 23:05:20.029204 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:20.029029 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="kube-rbac-proxy-web" containerID="cri-o://3a42ebbcead5ddc022fe1e76d17484702625ea59ee53400e65c560a0ccc6d5a7" gracePeriod=120 Sep 09 23:05:20.029204 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:20.029051 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="prom-label-proxy" containerID="cri-o://733094299f8dfd71c1670b69f698cbdf4bee17bf24bc74de3b5b25f922de8d09" gracePeriod=120 Sep 09 23:05:20.029204 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:20.029089 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="config-reloader" containerID="cri-o://ffe965fdf35fa4d4650bb88d7666120cef8b70f1b1ab7c6727880ad066cc9236" gracePeriod=120 Sep 09 23:05:20.029204 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:20.029106 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="kube-rbac-proxy-metric" containerID="cri-o://8c2a8a87a34f4637d97c2950a66389b53a59205967ce35135d396c3f0359309c" gracePeriod=120 Sep 09 23:05:21.091336 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.091300 2578 generic.go:356] "Generic (PLEG): container finished" podID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerID="733094299f8dfd71c1670b69f698cbdf4bee17bf24bc74de3b5b25f922de8d09" exitCode=0 Sep 09 23:05:21.091336 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.091333 2578 generic.go:356] "Generic (PLEG): container finished" podID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerID="cec9bc1e520b652033ac8a90132cc465a399e7f3a08bf341e7b522394ee15b34" exitCode=0 Sep 09 23:05:21.091336 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.091339 2578 generic.go:356] "Generic (PLEG): container finished" podID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerID="ffe965fdf35fa4d4650bb88d7666120cef8b70f1b1ab7c6727880ad066cc9236" exitCode=0 Sep 09 23:05:21.091735 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.091348 2578 generic.go:356] "Generic (PLEG): container finished" podID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerID="ffd3c1864c6ee5bdf68f3619b4814fc1c173ce4febe3551c1f2386af2d17e9a3" exitCode=0 Sep 09 23:05:21.091735 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.091365 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f","Type":"ContainerDied","Data":"733094299f8dfd71c1670b69f698cbdf4bee17bf24bc74de3b5b25f922de8d09"} Sep 09 23:05:21.091735 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.091449 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f","Type":"ContainerDied","Data":"cec9bc1e520b652033ac8a90132cc465a399e7f3a08bf341e7b522394ee15b34"} Sep 09 23:05:21.091735 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.091475 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f","Type":"ContainerDied","Data":"ffe965fdf35fa4d4650bb88d7666120cef8b70f1b1ab7c6727880ad066cc9236"} Sep 09 23:05:21.091735 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.091499 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f","Type":"ContainerDied","Data":"ffd3c1864c6ee5bdf68f3619b4814fc1c173ce4febe3551c1f2386af2d17e9a3"} Sep 09 23:05:21.275007 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.274984 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:21.375527 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.375448 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtvqp\" (UniqueName: \"kubernetes.io/projected/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-kube-api-access-jtvqp\") pod \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " Sep 09 23:05:21.375527 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.375489 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-main-tls\") pod \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " Sep 09 23:05:21.375527 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.375517 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-web-config\") pod \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " Sep 09 23:05:21.375787 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.375559 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " Sep 09 23:05:21.375787 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.375586 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-alertmanager-main-db\") pod \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " Sep 09 23:05:21.375787 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.375615 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-alertmanager-trusted-ca-bundle\") pod \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " Sep 09 23:05:21.375787 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.375639 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-cluster-tls-config\") pod \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " Sep 09 23:05:21.375787 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.375672 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-kube-rbac-proxy-web\") pod \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " Sep 09 23:05:21.375787 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.375708 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-config-out\") pod \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " Sep 09 23:05:21.375787 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.375730 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-config-volume\") pod \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " Sep 09 23:05:21.375787 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.375757 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-tls-assets\") pod \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " Sep 09 23:05:21.375787 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.375790 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-kube-rbac-proxy\") pod \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " Sep 09 23:05:21.376206 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.375815 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-metrics-client-ca\") pod \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\" (UID: \"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f\") " Sep 09 23:05:21.376206 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.375939 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" (UID: "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:05:21.376206 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.376009 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" (UID: "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:05:21.376206 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.376099 2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-alertmanager-main-db\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:05:21.376206 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.376120 2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:05:21.376500 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.376473 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" (UID: "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:05:21.378841 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.378583 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" (UID: "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:05:21.378975 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.378878 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-config-out" (OuterVolumeSpecName: "config-out") pod "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" (UID: "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:05:21.379037 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.379006 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" (UID: "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:05:21.379209 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.379175 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" (UID: "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:05:21.379209 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.379178 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" (UID: "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:05:21.379682 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.379649 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-kube-api-access-jtvqp" (OuterVolumeSpecName: "kube-api-access-jtvqp") pod "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" (UID: "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f"). InnerVolumeSpecName "kube-api-access-jtvqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:05:21.380513 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.380487 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" (UID: "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:05:21.381158 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.381138 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-config-volume" (OuterVolumeSpecName: "config-volume") pod "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" (UID: "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:05:21.382946 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.382927 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" (UID: "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:05:21.389238 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.389217 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-web-config" (OuterVolumeSpecName: "web-config") pod "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" (UID: "e8ae840a-52ae-4e68-8b87-a5b48b9aab6f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:05:21.477345 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.477310 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:05:21.477345 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.477339 2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-config-out\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:05:21.477345 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.477348 2578 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-config-volume\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:05:21.477570 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.477356 2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-tls-assets\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:05:21.477570 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.477366 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:05:21.477570 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.477374 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-metrics-client-ca\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:05:21.477570 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.477401 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtvqp\" (UniqueName: \"kubernetes.io/projected/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-kube-api-access-jtvqp\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:05:21.477570 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.477410 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-main-tls\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:05:21.477570 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.477418 2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-web-config\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:05:21.477570 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.477426 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:05:21.477570 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:21.477435 2578 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f-cluster-tls-config\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:05:22.098476 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.098446 2578 generic.go:356] "Generic (PLEG): container finished" podID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerID="8c2a8a87a34f4637d97c2950a66389b53a59205967ce35135d396c3f0359309c" exitCode=0 Sep 09 23:05:22.098476 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.098471 2578 generic.go:356] "Generic (PLEG): container finished" podID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerID="3a42ebbcead5ddc022fe1e76d17484702625ea59ee53400e65c560a0ccc6d5a7" exitCode=0 Sep 09 23:05:22.098850 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.098536 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.098850 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.098535 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f","Type":"ContainerDied","Data":"8c2a8a87a34f4637d97c2950a66389b53a59205967ce35135d396c3f0359309c"} Sep 09 23:05:22.098850 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.098606 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f","Type":"ContainerDied","Data":"3a42ebbcead5ddc022fe1e76d17484702625ea59ee53400e65c560a0ccc6d5a7"} Sep 09 23:05:22.098850 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.098635 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8ae840a-52ae-4e68-8b87-a5b48b9aab6f","Type":"ContainerDied","Data":"c85041c6764d48e6b1e58985fd3de86da1578fc090adbad1f58a6bb4db6b42d3"} Sep 09 23:05:22.098850 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.098666 2578 scope.go:117] "RemoveContainer" containerID="733094299f8dfd71c1670b69f698cbdf4bee17bf24bc74de3b5b25f922de8d09" Sep 09 23:05:22.107226 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.107201 2578 scope.go:117] "RemoveContainer" containerID="8c2a8a87a34f4637d97c2950a66389b53a59205967ce35135d396c3f0359309c" Sep 09 23:05:22.114162 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.114145 2578 scope.go:117] "RemoveContainer" containerID="cec9bc1e520b652033ac8a90132cc465a399e7f3a08bf341e7b522394ee15b34" Sep 09 23:05:22.118618 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.118597 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 23:05:22.121320 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.121298 2578 scope.go:117] "RemoveContainer" containerID="3a42ebbcead5ddc022fe1e76d17484702625ea59ee53400e65c560a0ccc6d5a7" Sep 09 23:05:22.122397 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.122359 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 23:05:22.127818 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.127803 2578 scope.go:117] "RemoveContainer" containerID="ffe965fdf35fa4d4650bb88d7666120cef8b70f1b1ab7c6727880ad066cc9236" Sep 09 23:05:22.134316 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.134301 2578 scope.go:117] "RemoveContainer" containerID="ffd3c1864c6ee5bdf68f3619b4814fc1c173ce4febe3551c1f2386af2d17e9a3" Sep 09 23:05:22.141263 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.141136 2578 scope.go:117] "RemoveContainer" containerID="611f1147ca45486c633f1d4e47137a46ad01d1027cd6b9d08eaedeb044ed917e" Sep 09 23:05:22.147531 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.147515 2578 scope.go:117] "RemoveContainer" containerID="733094299f8dfd71c1670b69f698cbdf4bee17bf24bc74de3b5b25f922de8d09" Sep 09 23:05:22.147788 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:05:22.147771 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"733094299f8dfd71c1670b69f698cbdf4bee17bf24bc74de3b5b25f922de8d09\": container with ID starting with 733094299f8dfd71c1670b69f698cbdf4bee17bf24bc74de3b5b25f922de8d09 not found: ID does not exist" containerID="733094299f8dfd71c1670b69f698cbdf4bee17bf24bc74de3b5b25f922de8d09" Sep 09 23:05:22.147832 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.147796 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"733094299f8dfd71c1670b69f698cbdf4bee17bf24bc74de3b5b25f922de8d09"} err="failed to get container status \"733094299f8dfd71c1670b69f698cbdf4bee17bf24bc74de3b5b25f922de8d09\": rpc error: code = NotFound desc = could not find container \"733094299f8dfd71c1670b69f698cbdf4bee17bf24bc74de3b5b25f922de8d09\": container with ID starting with 733094299f8dfd71c1670b69f698cbdf4bee17bf24bc74de3b5b25f922de8d09 not found: ID does not exist" Sep 09 23:05:22.147832 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.147813 2578 scope.go:117] "RemoveContainer" containerID="8c2a8a87a34f4637d97c2950a66389b53a59205967ce35135d396c3f0359309c" Sep 09 23:05:22.148051 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:05:22.148036 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c2a8a87a34f4637d97c2950a66389b53a59205967ce35135d396c3f0359309c\": container with ID starting with 8c2a8a87a34f4637d97c2950a66389b53a59205967ce35135d396c3f0359309c not found: ID does not exist" containerID="8c2a8a87a34f4637d97c2950a66389b53a59205967ce35135d396c3f0359309c" Sep 09 23:05:22.148090 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.148058 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c2a8a87a34f4637d97c2950a66389b53a59205967ce35135d396c3f0359309c"} err="failed to get container status \"8c2a8a87a34f4637d97c2950a66389b53a59205967ce35135d396c3f0359309c\": rpc error: code = NotFound desc = could not find container \"8c2a8a87a34f4637d97c2950a66389b53a59205967ce35135d396c3f0359309c\": container with ID starting with 8c2a8a87a34f4637d97c2950a66389b53a59205967ce35135d396c3f0359309c not found: ID does not exist" Sep 09 23:05:22.148090 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.148074 2578 scope.go:117] "RemoveContainer" containerID="cec9bc1e520b652033ac8a90132cc465a399e7f3a08bf341e7b522394ee15b34" Sep 09 23:05:22.148276 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:05:22.148260 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cec9bc1e520b652033ac8a90132cc465a399e7f3a08bf341e7b522394ee15b34\": container with ID starting with cec9bc1e520b652033ac8a90132cc465a399e7f3a08bf341e7b522394ee15b34 not found: ID does not exist" containerID="cec9bc1e520b652033ac8a90132cc465a399e7f3a08bf341e7b522394ee15b34" Sep 09 23:05:22.148316 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.148280 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cec9bc1e520b652033ac8a90132cc465a399e7f3a08bf341e7b522394ee15b34"} err="failed to get container status \"cec9bc1e520b652033ac8a90132cc465a399e7f3a08bf341e7b522394ee15b34\": rpc error: code = NotFound desc = could not find container \"cec9bc1e520b652033ac8a90132cc465a399e7f3a08bf341e7b522394ee15b34\": container with ID starting with cec9bc1e520b652033ac8a90132cc465a399e7f3a08bf341e7b522394ee15b34 not found: ID does not exist" Sep 09 23:05:22.148316 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.148294 2578 scope.go:117] "RemoveContainer" containerID="3a42ebbcead5ddc022fe1e76d17484702625ea59ee53400e65c560a0ccc6d5a7" Sep 09 23:05:22.148606 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:05:22.148585 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a42ebbcead5ddc022fe1e76d17484702625ea59ee53400e65c560a0ccc6d5a7\": container with ID starting with 3a42ebbcead5ddc022fe1e76d17484702625ea59ee53400e65c560a0ccc6d5a7 not found: ID does not exist" containerID="3a42ebbcead5ddc022fe1e76d17484702625ea59ee53400e65c560a0ccc6d5a7" Sep 09 23:05:22.148755 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.148611 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a42ebbcead5ddc022fe1e76d17484702625ea59ee53400e65c560a0ccc6d5a7"} err="failed to get container status \"3a42ebbcead5ddc022fe1e76d17484702625ea59ee53400e65c560a0ccc6d5a7\": rpc error: code = NotFound desc = could not find container \"3a42ebbcead5ddc022fe1e76d17484702625ea59ee53400e65c560a0ccc6d5a7\": container with ID starting with 3a42ebbcead5ddc022fe1e76d17484702625ea59ee53400e65c560a0ccc6d5a7 not found: ID does not exist" Sep 09 23:05:22.148755 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.148630 2578 scope.go:117] "RemoveContainer" containerID="ffe965fdf35fa4d4650bb88d7666120cef8b70f1b1ab7c6727880ad066cc9236" Sep 09 23:05:22.148959 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:05:22.148938 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffe965fdf35fa4d4650bb88d7666120cef8b70f1b1ab7c6727880ad066cc9236\": container with ID starting with ffe965fdf35fa4d4650bb88d7666120cef8b70f1b1ab7c6727880ad066cc9236 not found: ID does not exist" containerID="ffe965fdf35fa4d4650bb88d7666120cef8b70f1b1ab7c6727880ad066cc9236" Sep 09 23:05:22.149037 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.148965 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffe965fdf35fa4d4650bb88d7666120cef8b70f1b1ab7c6727880ad066cc9236"} err="failed to get container status \"ffe965fdf35fa4d4650bb88d7666120cef8b70f1b1ab7c6727880ad066cc9236\": rpc error: code = NotFound desc = could not find container \"ffe965fdf35fa4d4650bb88d7666120cef8b70f1b1ab7c6727880ad066cc9236\": container with ID starting with ffe965fdf35fa4d4650bb88d7666120cef8b70f1b1ab7c6727880ad066cc9236 not found: ID does not exist" Sep 09 23:05:22.149037 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.148987 2578 scope.go:117] "RemoveContainer" containerID="ffd3c1864c6ee5bdf68f3619b4814fc1c173ce4febe3551c1f2386af2d17e9a3" Sep 09 23:05:22.149308 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:05:22.149273 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffd3c1864c6ee5bdf68f3619b4814fc1c173ce4febe3551c1f2386af2d17e9a3\": container with ID starting with ffd3c1864c6ee5bdf68f3619b4814fc1c173ce4febe3551c1f2386af2d17e9a3 not found: ID does not exist" containerID="ffd3c1864c6ee5bdf68f3619b4814fc1c173ce4febe3551c1f2386af2d17e9a3" Sep 09 23:05:22.149308 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.149301 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffd3c1864c6ee5bdf68f3619b4814fc1c173ce4febe3551c1f2386af2d17e9a3"} err="failed to get container status \"ffd3c1864c6ee5bdf68f3619b4814fc1c173ce4febe3551c1f2386af2d17e9a3\": rpc error: code = NotFound desc = could not find container \"ffd3c1864c6ee5bdf68f3619b4814fc1c173ce4febe3551c1f2386af2d17e9a3\": container with ID starting with ffd3c1864c6ee5bdf68f3619b4814fc1c173ce4febe3551c1f2386af2d17e9a3 not found: ID does not exist" Sep 09 23:05:22.149465 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.149317 2578 scope.go:117] "RemoveContainer" containerID="611f1147ca45486c633f1d4e47137a46ad01d1027cd6b9d08eaedeb044ed917e" Sep 09 23:05:22.149628 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:05:22.149601 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"611f1147ca45486c633f1d4e47137a46ad01d1027cd6b9d08eaedeb044ed917e\": container with ID starting with 611f1147ca45486c633f1d4e47137a46ad01d1027cd6b9d08eaedeb044ed917e not found: ID does not exist" containerID="611f1147ca45486c633f1d4e47137a46ad01d1027cd6b9d08eaedeb044ed917e" Sep 09 23:05:22.149675 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.149633 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"611f1147ca45486c633f1d4e47137a46ad01d1027cd6b9d08eaedeb044ed917e"} err="failed to get container status \"611f1147ca45486c633f1d4e47137a46ad01d1027cd6b9d08eaedeb044ed917e\": rpc error: code = NotFound desc = could not find container \"611f1147ca45486c633f1d4e47137a46ad01d1027cd6b9d08eaedeb044ed917e\": container with ID starting with 611f1147ca45486c633f1d4e47137a46ad01d1027cd6b9d08eaedeb044ed917e not found: ID does not exist" Sep 09 23:05:22.149675 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.149646 2578 scope.go:117] "RemoveContainer" containerID="733094299f8dfd71c1670b69f698cbdf4bee17bf24bc74de3b5b25f922de8d09" Sep 09 23:05:22.149884 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.149865 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"733094299f8dfd71c1670b69f698cbdf4bee17bf24bc74de3b5b25f922de8d09"} err="failed to get container status \"733094299f8dfd71c1670b69f698cbdf4bee17bf24bc74de3b5b25f922de8d09\": rpc error: code = NotFound desc = could not find container \"733094299f8dfd71c1670b69f698cbdf4bee17bf24bc74de3b5b25f922de8d09\": container with ID starting with 733094299f8dfd71c1670b69f698cbdf4bee17bf24bc74de3b5b25f922de8d09 not found: ID does not exist" Sep 09 23:05:22.149954 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.149886 2578 scope.go:117] "RemoveContainer" containerID="8c2a8a87a34f4637d97c2950a66389b53a59205967ce35135d396c3f0359309c" Sep 09 23:05:22.150163 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150138 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c2a8a87a34f4637d97c2950a66389b53a59205967ce35135d396c3f0359309c"} err="failed to get container status \"8c2a8a87a34f4637d97c2950a66389b53a59205967ce35135d396c3f0359309c\": rpc error: code = NotFound desc = could not find container \"8c2a8a87a34f4637d97c2950a66389b53a59205967ce35135d396c3f0359309c\": container with ID starting with 8c2a8a87a34f4637d97c2950a66389b53a59205967ce35135d396c3f0359309c not found: ID does not exist" Sep 09 23:05:22.150211 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150166 2578 scope.go:117] "RemoveContainer" containerID="cec9bc1e520b652033ac8a90132cc465a399e7f3a08bf341e7b522394ee15b34" Sep 09 23:05:22.150262 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150247 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 23:05:22.150429 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150411 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cec9bc1e520b652033ac8a90132cc465a399e7f3a08bf341e7b522394ee15b34"} err="failed to get container status \"cec9bc1e520b652033ac8a90132cc465a399e7f3a08bf341e7b522394ee15b34\": rpc error: code = NotFound desc = could not find container \"cec9bc1e520b652033ac8a90132cc465a399e7f3a08bf341e7b522394ee15b34\": container with ID starting with cec9bc1e520b652033ac8a90132cc465a399e7f3a08bf341e7b522394ee15b34 not found: ID does not exist" Sep 09 23:05:22.150484 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150430 2578 scope.go:117] "RemoveContainer" containerID="3a42ebbcead5ddc022fe1e76d17484702625ea59ee53400e65c560a0ccc6d5a7" Sep 09 23:05:22.150540 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150528 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="init-config-reloader" Sep 09 23:05:22.150589 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150543 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="init-config-reloader" Sep 09 23:05:22.150589 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150551 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="kube-rbac-proxy-metric" Sep 09 23:05:22.150589 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150557 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="kube-rbac-proxy-metric" Sep 09 23:05:22.150589 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150568 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="prom-label-proxy" Sep 09 23:05:22.150589 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150576 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="prom-label-proxy" Sep 09 23:05:22.150589 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150584 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74fdfad4-a21b-45a2-9730-cfe25d4156c4" containerName="registry" Sep 09 23:05:22.150589 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150589 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="74fdfad4-a21b-45a2-9730-cfe25d4156c4" containerName="registry" Sep 09 23:05:22.150830 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150597 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="kube-rbac-proxy-web" Sep 09 23:05:22.150830 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150602 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="kube-rbac-proxy-web" Sep 09 23:05:22.150830 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150608 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="kube-rbac-proxy" Sep 09 23:05:22.150830 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150613 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="kube-rbac-proxy" Sep 09 23:05:22.150830 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150632 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="config-reloader" Sep 09 23:05:22.150830 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150637 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="config-reloader" Sep 09 23:05:22.150830 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150644 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="alertmanager" Sep 09 23:05:22.150830 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150649 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="alertmanager" Sep 09 23:05:22.150830 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150646 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a42ebbcead5ddc022fe1e76d17484702625ea59ee53400e65c560a0ccc6d5a7"} err="failed to get container status \"3a42ebbcead5ddc022fe1e76d17484702625ea59ee53400e65c560a0ccc6d5a7\": rpc error: code = NotFound desc = could not find container \"3a42ebbcead5ddc022fe1e76d17484702625ea59ee53400e65c560a0ccc6d5a7\": container with ID starting with 3a42ebbcead5ddc022fe1e76d17484702625ea59ee53400e65c560a0ccc6d5a7 not found: ID does not exist" Sep 09 23:05:22.150830 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150664 2578 scope.go:117] "RemoveContainer" containerID="ffe965fdf35fa4d4650bb88d7666120cef8b70f1b1ab7c6727880ad066cc9236" Sep 09 23:05:22.150830 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150693 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="74fdfad4-a21b-45a2-9730-cfe25d4156c4" containerName="registry" Sep 09 23:05:22.150830 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150701 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="kube-rbac-proxy-web" Sep 09 23:05:22.150830 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150706 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="kube-rbac-proxy" Sep 09 23:05:22.150830 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150712 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="prom-label-proxy" Sep 09 23:05:22.150830 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150718 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="config-reloader" Sep 09 23:05:22.150830 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150723 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="alertmanager" Sep 09 23:05:22.150830 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150731 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" containerName="kube-rbac-proxy-metric" Sep 09 23:05:22.151465 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150875 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffe965fdf35fa4d4650bb88d7666120cef8b70f1b1ab7c6727880ad066cc9236"} err="failed to get container status \"ffe965fdf35fa4d4650bb88d7666120cef8b70f1b1ab7c6727880ad066cc9236\": rpc error: code = NotFound desc = could not find container \"ffe965fdf35fa4d4650bb88d7666120cef8b70f1b1ab7c6727880ad066cc9236\": container with ID starting with ffe965fdf35fa4d4650bb88d7666120cef8b70f1b1ab7c6727880ad066cc9236 not found: ID does not exist" Sep 09 23:05:22.151465 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.150897 2578 scope.go:117] "RemoveContainer" containerID="ffd3c1864c6ee5bdf68f3619b4814fc1c173ce4febe3551c1f2386af2d17e9a3" Sep 09 23:05:22.151465 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.151107 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffd3c1864c6ee5bdf68f3619b4814fc1c173ce4febe3551c1f2386af2d17e9a3"} err="failed to get container status \"ffd3c1864c6ee5bdf68f3619b4814fc1c173ce4febe3551c1f2386af2d17e9a3\": rpc error: code = NotFound desc = could not find container \"ffd3c1864c6ee5bdf68f3619b4814fc1c173ce4febe3551c1f2386af2d17e9a3\": container with ID starting with ffd3c1864c6ee5bdf68f3619b4814fc1c173ce4febe3551c1f2386af2d17e9a3 not found: ID does not exist" Sep 09 23:05:22.151465 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.151125 2578 scope.go:117] "RemoveContainer" containerID="611f1147ca45486c633f1d4e47137a46ad01d1027cd6b9d08eaedeb044ed917e" Sep 09 23:05:22.151465 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.151343 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"611f1147ca45486c633f1d4e47137a46ad01d1027cd6b9d08eaedeb044ed917e"} err="failed to get container status \"611f1147ca45486c633f1d4e47137a46ad01d1027cd6b9d08eaedeb044ed917e\": rpc error: code = NotFound desc = could not find container \"611f1147ca45486c633f1d4e47137a46ad01d1027cd6b9d08eaedeb044ed917e\": container with ID starting with 611f1147ca45486c633f1d4e47137a46ad01d1027cd6b9d08eaedeb044ed917e not found: ID does not exist" Sep 09 23:05:22.155614 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.155599 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.157970 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.157936 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 09 23:05:22.157970 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.157965 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 09 23:05:22.158157 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.158100 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 09 23:05:22.158157 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.158105 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 09 23:05:22.158289 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.158188 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 09 23:05:22.158332 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.158290 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 09 23:05:22.158437 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.158342 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4v4r9\"" Sep 09 23:05:22.158437 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.158417 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 09 23:05:22.158556 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.158503 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 09 23:05:22.163062 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.163036 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 09 23:05:22.168285 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.168267 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 23:05:22.282830 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.282799 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-config-volume\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.282830 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.282829 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.283022 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.282859 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.283022 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.282912 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmgs8\" (UniqueName: \"kubernetes.io/projected/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-kube-api-access-tmgs8\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.283022 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.282952 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.283022 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.282978 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.283022 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.282992 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.283022 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.283008 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-web-config\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.283199 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.283034 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.283199 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.283055 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-config-out\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.283199 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.283090 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.283199 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.283131 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.283199 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.283156 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.384209 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.384137 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.384209 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.384172 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-config-out\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.384209 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.384188 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.384456 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.384307 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.384456 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.384345 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.384456 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.384370 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-config-volume\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.384456 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.384417 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.384685 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.384472 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.384685 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.384501 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmgs8\" (UniqueName: \"kubernetes.io/projected/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-kube-api-access-tmgs8\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.384685 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.384549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.384685 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.384570 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.384685 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.384594 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.384685 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.384624 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-web-config\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.384685 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.384655 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.385770 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.385230 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.385770 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.385245 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.387294 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.387270 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-config-out\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.387717 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.387655 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.388284 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.388031 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.388284 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.388043 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.388284 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.388223 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-web-config\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.388284 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.388228 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.388611 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.388592 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.388788 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.388767 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-config-volume\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.389840 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.389824 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.394149 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.394125 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmgs8\" (UniqueName: \"kubernetes.io/projected/3fbde13f-9319-4ad4-acf8-a2eb971ff8b6-kube-api-access-tmgs8\") pod \"alertmanager-main-0\" (UID: \"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.465358 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.465331 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 23:05:22.763231 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:22.763198 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 23:05:22.766578 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:05:22.766536 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbde13f_9319_4ad4_acf8_a2eb971ff8b6.slice/crio-326293745a1f23132a652f4e7386c3f14483d9e2813cfcb800d0e0acff66b698 WatchSource:0}: Error finding container 326293745a1f23132a652f4e7386c3f14483d9e2813cfcb800d0e0acff66b698: Status 404 returned error can't find the container with id 326293745a1f23132a652f4e7386c3f14483d9e2813cfcb800d0e0acff66b698 Sep 09 23:05:23.105759 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:23.105663 2578 generic.go:356] "Generic (PLEG): container finished" podID="3fbde13f-9319-4ad4-acf8-a2eb971ff8b6" containerID="e1f04137f2b2014c09c0d1e55f426ff273cf09e6d512e3aa69dd7d34d0356c73" exitCode=0 Sep 09 23:05:23.105759 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:23.105725 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6","Type":"ContainerDied","Data":"e1f04137f2b2014c09c0d1e55f426ff273cf09e6d512e3aa69dd7d34d0356c73"} Sep 09 23:05:23.106161 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:23.105788 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6","Type":"ContainerStarted","Data":"326293745a1f23132a652f4e7386c3f14483d9e2813cfcb800d0e0acff66b698"} Sep 09 23:05:23.851069 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:23.851026 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8ae840a-52ae-4e68-8b87-a5b48b9aab6f" path="/var/lib/kubelet/pods/e8ae840a-52ae-4e68-8b87-a5b48b9aab6f/volumes" Sep 09 23:05:24.050193 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.050151 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-74d5594d54-qlf7h"] Sep 09 23:05:24.053773 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.053754 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.056041 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.056020 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 09 23:05:24.056239 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.056200 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-kntcq\"" Sep 09 23:05:24.056342 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.056232 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 09 23:05:24.056342 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.056247 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 09 23:05:24.056342 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.056206 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 09 23:05:24.056736 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.056721 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 09 23:05:24.060991 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.060976 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 09 23:05:24.068762 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.068740 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-74d5594d54-qlf7h"] Sep 09 23:05:24.114070 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.113998 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6","Type":"ContainerStarted","Data":"43b19a6fa026be9831cbbb13164306024d1466f42c4f330ad2eafa5d8af19461"} Sep 09 23:05:24.114070 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.114056 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6","Type":"ContainerStarted","Data":"bdff745bf84be0942a2b78e5e7b9ae4eea6c09272c58432f5f25bbeee3f66729"} Sep 09 23:05:24.114456 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.114080 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6","Type":"ContainerStarted","Data":"c54ec98c5b427570f0b7d624d74ab6f231fbb5d9f74deb0f5890a4c250321c02"} Sep 09 23:05:24.114456 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.114104 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6","Type":"ContainerStarted","Data":"d651a869355ac8ebd6351d0eab19148c4da69f387e72a451ddd130b5da90a82d"} Sep 09 23:05:24.114456 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.114132 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6","Type":"ContainerStarted","Data":"73a25e944f263f540776b64433b7825baaf654847e7c822908f0faf3c1eb5b0f"} Sep 09 23:05:24.114456 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.114158 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fbde13f-9319-4ad4-acf8-a2eb971ff8b6","Type":"ContainerStarted","Data":"e073e62ee80012a5519d480a7d541edfc43de6f3870a86ccab2db0c09dcf749e"} Sep 09 23:05:24.141061 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.141005 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.125343763 podStartE2EDuration="2.140988589s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.644826ms" isStatefulPod=false podCreationTimestamp="2026-09-09 23:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:05:24.140006192 +0000 UTC m=+384.890591130" watchObservedRunningTime="2026-09-09 23:05:24.140988589 +0000 UTC m=+384.891573516" Sep 09 23:05:24.198702 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.198656 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2c4dbc94-e815-4e1c-ba62-bd74df448ca5-secret-telemeter-client\") pod \"telemeter-client-74d5594d54-qlf7h\" (UID: \"2c4dbc94-e815-4e1c-ba62-bd74df448ca5\") " pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.199087 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.198716 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c4dbc94-e815-4e1c-ba62-bd74df448ca5-metrics-client-ca\") pod \"telemeter-client-74d5594d54-qlf7h\" (UID: \"2c4dbc94-e815-4e1c-ba62-bd74df448ca5\") " pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.199087 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.198764 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c4dbc94-e815-4e1c-ba62-bd74df448ca5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74d5594d54-qlf7h\" (UID: \"2c4dbc94-e815-4e1c-ba62-bd74df448ca5\") " pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.199087 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.198792 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdr22\" (UniqueName: \"kubernetes.io/projected/2c4dbc94-e815-4e1c-ba62-bd74df448ca5-kube-api-access-mdr22\") pod \"telemeter-client-74d5594d54-qlf7h\" (UID: \"2c4dbc94-e815-4e1c-ba62-bd74df448ca5\") " pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.199087 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.198826 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c4dbc94-e815-4e1c-ba62-bd74df448ca5-serving-certs-ca-bundle\") pod \"telemeter-client-74d5594d54-qlf7h\" (UID: \"2c4dbc94-e815-4e1c-ba62-bd74df448ca5\") " pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.199087 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.198863 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2c4dbc94-e815-4e1c-ba62-bd74df448ca5-federate-client-tls\") pod \"telemeter-client-74d5594d54-qlf7h\" (UID: \"2c4dbc94-e815-4e1c-ba62-bd74df448ca5\") " pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.199087 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.198918 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c4dbc94-e815-4e1c-ba62-bd74df448ca5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74d5594d54-qlf7h\" (UID: \"2c4dbc94-e815-4e1c-ba62-bd74df448ca5\") " pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.199087 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.198954 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2c4dbc94-e815-4e1c-ba62-bd74df448ca5-telemeter-client-tls\") pod \"telemeter-client-74d5594d54-qlf7h\" (UID: \"2c4dbc94-e815-4e1c-ba62-bd74df448ca5\") " pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.300277 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.300243 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c4dbc94-e815-4e1c-ba62-bd74df448ca5-serving-certs-ca-bundle\") pod \"telemeter-client-74d5594d54-qlf7h\" (UID: \"2c4dbc94-e815-4e1c-ba62-bd74df448ca5\") " pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.300460 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.300420 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2c4dbc94-e815-4e1c-ba62-bd74df448ca5-federate-client-tls\") pod \"telemeter-client-74d5594d54-qlf7h\" (UID: \"2c4dbc94-e815-4e1c-ba62-bd74df448ca5\") " pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.300548 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.300481 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c4dbc94-e815-4e1c-ba62-bd74df448ca5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74d5594d54-qlf7h\" (UID: \"2c4dbc94-e815-4e1c-ba62-bd74df448ca5\") " pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.300548 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.300538 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2c4dbc94-e815-4e1c-ba62-bd74df448ca5-telemeter-client-tls\") pod \"telemeter-client-74d5594d54-qlf7h\" (UID: \"2c4dbc94-e815-4e1c-ba62-bd74df448ca5\") " pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.300658 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.300635 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2c4dbc94-e815-4e1c-ba62-bd74df448ca5-secret-telemeter-client\") pod \"telemeter-client-74d5594d54-qlf7h\" (UID: \"2c4dbc94-e815-4e1c-ba62-bd74df448ca5\") " pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.300729 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.300712 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c4dbc94-e815-4e1c-ba62-bd74df448ca5-metrics-client-ca\") pod \"telemeter-client-74d5594d54-qlf7h\" (UID: \"2c4dbc94-e815-4e1c-ba62-bd74df448ca5\") " pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.301539 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.300795 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c4dbc94-e815-4e1c-ba62-bd74df448ca5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74d5594d54-qlf7h\" (UID: \"2c4dbc94-e815-4e1c-ba62-bd74df448ca5\") " pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.301539 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.300872 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdr22\" (UniqueName: \"kubernetes.io/projected/2c4dbc94-e815-4e1c-ba62-bd74df448ca5-kube-api-access-mdr22\") pod \"telemeter-client-74d5594d54-qlf7h\" (UID: \"2c4dbc94-e815-4e1c-ba62-bd74df448ca5\") " pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.301539 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.301024 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c4dbc94-e815-4e1c-ba62-bd74df448ca5-serving-certs-ca-bundle\") pod \"telemeter-client-74d5594d54-qlf7h\" (UID: \"2c4dbc94-e815-4e1c-ba62-bd74df448ca5\") " pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.301539 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.301479 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c4dbc94-e815-4e1c-ba62-bd74df448ca5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74d5594d54-qlf7h\" (UID: \"2c4dbc94-e815-4e1c-ba62-bd74df448ca5\") " pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.301862 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.301836 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c4dbc94-e815-4e1c-ba62-bd74df448ca5-metrics-client-ca\") pod \"telemeter-client-74d5594d54-qlf7h\" (UID: \"2c4dbc94-e815-4e1c-ba62-bd74df448ca5\") " pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.303325 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.303304 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c4dbc94-e815-4e1c-ba62-bd74df448ca5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74d5594d54-qlf7h\" (UID: \"2c4dbc94-e815-4e1c-ba62-bd74df448ca5\") " pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.303443 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.303329 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2c4dbc94-e815-4e1c-ba62-bd74df448ca5-telemeter-client-tls\") pod \"telemeter-client-74d5594d54-qlf7h\" (UID: \"2c4dbc94-e815-4e1c-ba62-bd74df448ca5\") " pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.303483 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.303447 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2c4dbc94-e815-4e1c-ba62-bd74df448ca5-federate-client-tls\") pod \"telemeter-client-74d5594d54-qlf7h\" (UID: \"2c4dbc94-e815-4e1c-ba62-bd74df448ca5\") " pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.303483 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.303455 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2c4dbc94-e815-4e1c-ba62-bd74df448ca5-secret-telemeter-client\") pod \"telemeter-client-74d5594d54-qlf7h\" (UID: \"2c4dbc94-e815-4e1c-ba62-bd74df448ca5\") " pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.308678 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.308656 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdr22\" (UniqueName: \"kubernetes.io/projected/2c4dbc94-e815-4e1c-ba62-bd74df448ca5-kube-api-access-mdr22\") pod \"telemeter-client-74d5594d54-qlf7h\" (UID: \"2c4dbc94-e815-4e1c-ba62-bd74df448ca5\") " pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.363156 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.363124 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" Sep 09 23:05:24.638454 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:24.638424 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-74d5594d54-qlf7h"] Sep 09 23:05:24.639903 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:05:24.639877 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c4dbc94_e815_4e1c_ba62_bd74df448ca5.slice/crio-75959c6959fc574a4ca545f0f951099980f42719ea0fe34fa0039993b6b22d3c WatchSource:0}: Error finding container 75959c6959fc574a4ca545f0f951099980f42719ea0fe34fa0039993b6b22d3c: Status 404 returned error can't find the container with id 75959c6959fc574a4ca545f0f951099980f42719ea0fe34fa0039993b6b22d3c Sep 09 23:05:25.119424 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:25.119393 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" event={"ID":"2c4dbc94-e815-4e1c-ba62-bd74df448ca5","Type":"ContainerStarted","Data":"75959c6959fc574a4ca545f0f951099980f42719ea0fe34fa0039993b6b22d3c"} Sep 09 23:05:27.135106 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:27.135003 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" event={"ID":"2c4dbc94-e815-4e1c-ba62-bd74df448ca5","Type":"ContainerStarted","Data":"04630774cffe6a59caa3eaee98f031215c22d8cea835bd268a85e97f4202673b"} Sep 09 23:05:27.135106 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:27.135100 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" event={"ID":"2c4dbc94-e815-4e1c-ba62-bd74df448ca5","Type":"ContainerStarted","Data":"c85cb55db7fe3c2603d84537de8d3c6a57ee3a2840e4032b29ded6e2aa3bd2c4"} Sep 09 23:05:27.135664 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:27.135128 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" event={"ID":"2c4dbc94-e815-4e1c-ba62-bd74df448ca5","Type":"ContainerStarted","Data":"93a224cc61584a7eacc43b8849374d564029387ef91dfa2ae15fc0a67cc4ebab"} Sep 09 23:05:27.156146 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:05:27.156091 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-74d5594d54-qlf7h" podStartSLOduration=1.019286576 podStartE2EDuration="3.156078172s" totalImagesPullingTime="2.136791596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:05:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:05:27.155966534 +0000 UTC m=+387.906551479" watchObservedRunningTime="2026-09-09 23:05:27.156078172 +0000 UTC m=+387.906663110" Sep 09 23:07:56.992518 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:07:56.992440 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j77n5"] Sep 09 23:07:56.995574 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:07:56.995558 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j77n5" Sep 09 23:07:56.998127 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:07:56.998103 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 09 23:07:56.998289 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:07:56.998160 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-bbghv\"" Sep 09 23:07:56.998289 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:07:56.998163 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 09 23:07:57.006853 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:07:57.006818 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j77n5"] Sep 09 23:07:57.069466 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:07:57.069432 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2646df1c-0461-406b-8c2c-ddaa6d702aba-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-j77n5\" (UID: \"2646df1c-0461-406b-8c2c-ddaa6d702aba\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j77n5" Sep 09 23:07:57.069649 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:07:57.069504 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7x78\" (UniqueName: \"kubernetes.io/projected/2646df1c-0461-406b-8c2c-ddaa6d702aba-kube-api-access-d7x78\") pod \"cert-manager-operator-controller-manager-55c444bf86-j77n5\" (UID: \"2646df1c-0461-406b-8c2c-ddaa6d702aba\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j77n5" Sep 09 23:07:57.170325 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:07:57.170286 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2646df1c-0461-406b-8c2c-ddaa6d702aba-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-j77n5\" (UID: \"2646df1c-0461-406b-8c2c-ddaa6d702aba\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j77n5" Sep 09 23:07:57.170511 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:07:57.170373 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7x78\" (UniqueName: \"kubernetes.io/projected/2646df1c-0461-406b-8c2c-ddaa6d702aba-kube-api-access-d7x78\") pod \"cert-manager-operator-controller-manager-55c444bf86-j77n5\" (UID: \"2646df1c-0461-406b-8c2c-ddaa6d702aba\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j77n5" Sep 09 23:07:57.170692 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:07:57.170673 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2646df1c-0461-406b-8c2c-ddaa6d702aba-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-j77n5\" (UID: \"2646df1c-0461-406b-8c2c-ddaa6d702aba\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j77n5" Sep 09 23:07:57.178624 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:07:57.178599 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7x78\" (UniqueName: \"kubernetes.io/projected/2646df1c-0461-406b-8c2c-ddaa6d702aba-kube-api-access-d7x78\") pod \"cert-manager-operator-controller-manager-55c444bf86-j77n5\" (UID: \"2646df1c-0461-406b-8c2c-ddaa6d702aba\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j77n5" Sep 09 23:07:57.304496 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:07:57.304471 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j77n5" Sep 09 23:07:57.580987 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:07:57.580857 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j77n5"] Sep 09 23:07:57.584719 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:07:57.584692 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2646df1c_0461_406b_8c2c_ddaa6d702aba.slice/crio-053419c55f1865de0fb6c92b4074d100f1ad4debd929bef13a3f9d56e6fa59bf WatchSource:0}: Error finding container 053419c55f1865de0fb6c92b4074d100f1ad4debd929bef13a3f9d56e6fa59bf: Status 404 returned error can't find the container with id 053419c55f1865de0fb6c92b4074d100f1ad4debd929bef13a3f9d56e6fa59bf Sep 09 23:07:57.687104 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:07:57.687071 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j77n5" event={"ID":"2646df1c-0461-406b-8c2c-ddaa6d702aba","Type":"ContainerStarted","Data":"053419c55f1865de0fb6c92b4074d100f1ad4debd929bef13a3f9d56e6fa59bf"} Sep 09 23:08:00.701803 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:00.701767 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j77n5" event={"ID":"2646df1c-0461-406b-8c2c-ddaa6d702aba","Type":"ContainerStarted","Data":"db7e68f382624dce3d12c279bc9c8e2a8cc9074ae244583258e3579853e4fa37"} Sep 09 23:08:00.722411 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:00.722344 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j77n5" podStartSLOduration=2.142762264 podStartE2EDuration="4.72232734s" totalImagesPullingTime="2.579565076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:07:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:08:00.720908346 +0000 UTC m=+541.471493298" watchObservedRunningTime="2026-09-09 23:08:00.72232734 +0000 UTC m=+541.472912267" Sep 09 23:08:01.282601 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:01.282570 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7hgdk"] Sep 09 23:08:01.285725 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:01.285708 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-7hgdk" Sep 09 23:08:01.288853 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:01.288828 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 09 23:08:01.288960 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:01.288924 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 09 23:08:01.289286 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:01.289272 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-zjxjv\"" Sep 09 23:08:01.305221 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:01.305192 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf3ff5f6-4e2c-4bfe-b5b5-f4db0d1e3ae5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7hgdk\" (UID: \"bf3ff5f6-4e2c-4bfe-b5b5-f4db0d1e3ae5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7hgdk" Sep 09 23:08:01.305353 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:01.305253 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqjlp\" (UniqueName: \"kubernetes.io/projected/bf3ff5f6-4e2c-4bfe-b5b5-f4db0d1e3ae5-kube-api-access-dqjlp\") pod \"cert-manager-webhook-6486ffd997-7hgdk\" (UID: \"bf3ff5f6-4e2c-4bfe-b5b5-f4db0d1e3ae5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7hgdk" Sep 09 23:08:01.310344 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:01.310320 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7hgdk"] Sep 09 23:08:01.405933 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:01.405890 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqjlp\" (UniqueName: \"kubernetes.io/projected/bf3ff5f6-4e2c-4bfe-b5b5-f4db0d1e3ae5-kube-api-access-dqjlp\") pod \"cert-manager-webhook-6486ffd997-7hgdk\" (UID: \"bf3ff5f6-4e2c-4bfe-b5b5-f4db0d1e3ae5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7hgdk" Sep 09 23:08:01.406099 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:01.405960 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf3ff5f6-4e2c-4bfe-b5b5-f4db0d1e3ae5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7hgdk\" (UID: \"bf3ff5f6-4e2c-4bfe-b5b5-f4db0d1e3ae5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7hgdk" Sep 09 23:08:01.419814 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:01.419780 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf3ff5f6-4e2c-4bfe-b5b5-f4db0d1e3ae5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7hgdk\" (UID: \"bf3ff5f6-4e2c-4bfe-b5b5-f4db0d1e3ae5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7hgdk" Sep 09 23:08:01.420019 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:01.419996 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqjlp\" (UniqueName: \"kubernetes.io/projected/bf3ff5f6-4e2c-4bfe-b5b5-f4db0d1e3ae5-kube-api-access-dqjlp\") pod \"cert-manager-webhook-6486ffd997-7hgdk\" (UID: \"bf3ff5f6-4e2c-4bfe-b5b5-f4db0d1e3ae5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7hgdk" Sep 09 23:08:01.594996 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:01.594912 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-7hgdk" Sep 09 23:08:01.874415 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:01.874314 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7hgdk"] Sep 09 23:08:01.877206 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:08:01.877180 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf3ff5f6_4e2c_4bfe_b5b5_f4db0d1e3ae5.slice/crio-65cd50dae729af592c6347d65fd2b5c1ec4b221e4bca0593e17727a4b97fe3a0 WatchSource:0}: Error finding container 65cd50dae729af592c6347d65fd2b5c1ec4b221e4bca0593e17727a4b97fe3a0: Status 404 returned error can't find the container with id 65cd50dae729af592c6347d65fd2b5c1ec4b221e4bca0593e17727a4b97fe3a0 Sep 09 23:08:02.712778 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:02.712738 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-7hgdk" event={"ID":"bf3ff5f6-4e2c-4bfe-b5b5-f4db0d1e3ae5","Type":"ContainerStarted","Data":"65cd50dae729af592c6347d65fd2b5c1ec4b221e4bca0593e17727a4b97fe3a0"} Sep 09 23:08:04.724193 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:04.724108 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-7hgdk" event={"ID":"bf3ff5f6-4e2c-4bfe-b5b5-f4db0d1e3ae5","Type":"ContainerStarted","Data":"7fad680aee920064a4a20d9f3538e09295937d5de02671609e984f537bd7794d"} Sep 09 23:08:04.724193 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:04.724185 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-7hgdk" Sep 09 23:08:04.739540 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:04.739494 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-7hgdk" podStartSLOduration=1.211115036 podStartE2EDuration="3.739479757s" totalImagesPullingTime="2.528364721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:08:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:08:04.738588554 +0000 UTC m=+545.489173494" watchObservedRunningTime="2026-09-09 23:08:04.739479757 +0000 UTC m=+545.490064682" Sep 09 23:08:10.729959 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:10.729925 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-7hgdk" Sep 09 23:08:12.502454 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:12.502419 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rfrl6"] Sep 09 23:08:12.505572 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:12.505554 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rfrl6" Sep 09 23:08:12.507895 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:12.507871 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-pm9x4\"" Sep 09 23:08:12.512911 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:12.512890 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rfrl6"] Sep 09 23:08:12.592577 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:12.592544 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8292e028-4368-4260-a4dd-ae5b3262255c-bound-sa-token\") pod \"cert-manager-55b4d954d5-rfrl6\" (UID: \"8292e028-4368-4260-a4dd-ae5b3262255c\") " pod="cert-manager/cert-manager-55b4d954d5-rfrl6" Sep 09 23:08:12.592717 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:12.592586 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9k6w\" (UniqueName: \"kubernetes.io/projected/8292e028-4368-4260-a4dd-ae5b3262255c-kube-api-access-h9k6w\") pod \"cert-manager-55b4d954d5-rfrl6\" (UID: \"8292e028-4368-4260-a4dd-ae5b3262255c\") " pod="cert-manager/cert-manager-55b4d954d5-rfrl6" Sep 09 23:08:12.693648 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:12.693609 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8292e028-4368-4260-a4dd-ae5b3262255c-bound-sa-token\") pod \"cert-manager-55b4d954d5-rfrl6\" (UID: \"8292e028-4368-4260-a4dd-ae5b3262255c\") " pod="cert-manager/cert-manager-55b4d954d5-rfrl6" Sep 09 23:08:12.693791 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:12.693660 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9k6w\" (UniqueName: \"kubernetes.io/projected/8292e028-4368-4260-a4dd-ae5b3262255c-kube-api-access-h9k6w\") pod \"cert-manager-55b4d954d5-rfrl6\" (UID: \"8292e028-4368-4260-a4dd-ae5b3262255c\") " pod="cert-manager/cert-manager-55b4d954d5-rfrl6" Sep 09 23:08:12.702995 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:12.702968 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8292e028-4368-4260-a4dd-ae5b3262255c-bound-sa-token\") pod \"cert-manager-55b4d954d5-rfrl6\" (UID: \"8292e028-4368-4260-a4dd-ae5b3262255c\") " pod="cert-manager/cert-manager-55b4d954d5-rfrl6" Sep 09 23:08:12.703110 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:12.703097 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9k6w\" (UniqueName: \"kubernetes.io/projected/8292e028-4368-4260-a4dd-ae5b3262255c-kube-api-access-h9k6w\") pod \"cert-manager-55b4d954d5-rfrl6\" (UID: \"8292e028-4368-4260-a4dd-ae5b3262255c\") " pod="cert-manager/cert-manager-55b4d954d5-rfrl6" Sep 09 23:08:12.815632 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:12.815610 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rfrl6" Sep 09 23:08:13.083372 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:13.083294 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rfrl6"] Sep 09 23:08:13.085962 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:08:13.085936 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8292e028_4368_4260_a4dd_ae5b3262255c.slice/crio-a4d687565d45c6d562cdaaf2e1e410a59698d29a81eb2934229cc9d3135f6215 WatchSource:0}: Error finding container a4d687565d45c6d562cdaaf2e1e410a59698d29a81eb2934229cc9d3135f6215: Status 404 returned error can't find the container with id a4d687565d45c6d562cdaaf2e1e410a59698d29a81eb2934229cc9d3135f6215 Sep 09 23:08:13.761421 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:13.761372 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rfrl6" event={"ID":"8292e028-4368-4260-a4dd-ae5b3262255c","Type":"ContainerStarted","Data":"1a2bd997bd84912dce5bc19ac4a777d91b98f58cff9febae9a52ef6124209019"} Sep 09 23:08:13.761967 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:13.761426 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rfrl6" event={"ID":"8292e028-4368-4260-a4dd-ae5b3262255c","Type":"ContainerStarted","Data":"a4d687565d45c6d562cdaaf2e1e410a59698d29a81eb2934229cc9d3135f6215"} Sep 09 23:08:13.777620 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:13.777576 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-rfrl6" podStartSLOduration=1.777560593 podStartE2EDuration="1.777560593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:08:13.776336356 +0000 UTC m=+554.526921282" watchObservedRunningTime="2026-09-09 23:08:13.777560593 +0000 UTC m=+554.528145520" Sep 09 23:08:48.927503 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:48.927471 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4"] Sep 09 23:08:48.932393 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:48.932355 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4" Sep 09 23:08:48.935148 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:48.935119 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 09 23:08:48.935282 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:48.935156 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 09 23:08:48.935463 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:48.935446 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vkrgg\"" Sep 09 23:08:48.935553 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:48.935500 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 09 23:08:48.935724 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:48.935708 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 09 23:08:48.949436 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:48.949415 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4"] Sep 09 23:08:48.984016 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:48.983987 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df244f8e-1b94-47ef-9c41-f0ce67302a5d-webhook-cert\") pod \"opendatahub-operator-controller-manager-65d8796fc4-9t9t4\" (UID: \"df244f8e-1b94-47ef-9c41-f0ce67302a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4" Sep 09 23:08:48.984150 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:48.984025 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbqr5\" (UniqueName: \"kubernetes.io/projected/df244f8e-1b94-47ef-9c41-f0ce67302a5d-kube-api-access-wbqr5\") pod \"opendatahub-operator-controller-manager-65d8796fc4-9t9t4\" (UID: \"df244f8e-1b94-47ef-9c41-f0ce67302a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4" Sep 09 23:08:48.984150 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:48.984089 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df244f8e-1b94-47ef-9c41-f0ce67302a5d-tmp\") pod \"opendatahub-operator-controller-manager-65d8796fc4-9t9t4\" (UID: \"df244f8e-1b94-47ef-9c41-f0ce67302a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4" Sep 09 23:08:48.984150 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:48.984117 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/df244f8e-1b94-47ef-9c41-f0ce67302a5d-manifests\") pod \"opendatahub-operator-controller-manager-65d8796fc4-9t9t4\" (UID: \"df244f8e-1b94-47ef-9c41-f0ce67302a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4" Sep 09 23:08:48.984284 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:48.984153 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/df244f8e-1b94-47ef-9c41-f0ce67302a5d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65d8796fc4-9t9t4\" (UID: \"df244f8e-1b94-47ef-9c41-f0ce67302a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4" Sep 09 23:08:49.084501 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:49.084466 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df244f8e-1b94-47ef-9c41-f0ce67302a5d-webhook-cert\") pod \"opendatahub-operator-controller-manager-65d8796fc4-9t9t4\" (UID: \"df244f8e-1b94-47ef-9c41-f0ce67302a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4" Sep 09 23:08:49.084501 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:49.084502 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbqr5\" (UniqueName: \"kubernetes.io/projected/df244f8e-1b94-47ef-9c41-f0ce67302a5d-kube-api-access-wbqr5\") pod \"opendatahub-operator-controller-manager-65d8796fc4-9t9t4\" (UID: \"df244f8e-1b94-47ef-9c41-f0ce67302a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4" Sep 09 23:08:49.084727 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:49.084534 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df244f8e-1b94-47ef-9c41-f0ce67302a5d-tmp\") pod \"opendatahub-operator-controller-manager-65d8796fc4-9t9t4\" (UID: \"df244f8e-1b94-47ef-9c41-f0ce67302a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4" Sep 09 23:08:49.084727 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:49.084623 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/df244f8e-1b94-47ef-9c41-f0ce67302a5d-manifests\") pod \"opendatahub-operator-controller-manager-65d8796fc4-9t9t4\" (UID: \"df244f8e-1b94-47ef-9c41-f0ce67302a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4" Sep 09 23:08:49.084727 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:49.084671 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/df244f8e-1b94-47ef-9c41-f0ce67302a5d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65d8796fc4-9t9t4\" (UID: \"df244f8e-1b94-47ef-9c41-f0ce67302a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4" Sep 09 23:08:49.084997 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:49.084976 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/df244f8e-1b94-47ef-9c41-f0ce67302a5d-manifests\") pod \"opendatahub-operator-controller-manager-65d8796fc4-9t9t4\" (UID: \"df244f8e-1b94-47ef-9c41-f0ce67302a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4" Sep 09 23:08:49.087037 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:49.087014 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df244f8e-1b94-47ef-9c41-f0ce67302a5d-tmp\") pod \"opendatahub-operator-controller-manager-65d8796fc4-9t9t4\" (UID: \"df244f8e-1b94-47ef-9c41-f0ce67302a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4" Sep 09 23:08:49.087175 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:49.087156 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df244f8e-1b94-47ef-9c41-f0ce67302a5d-webhook-cert\") pod \"opendatahub-operator-controller-manager-65d8796fc4-9t9t4\" (UID: \"df244f8e-1b94-47ef-9c41-f0ce67302a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4" Sep 09 23:08:49.087231 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:49.087177 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/df244f8e-1b94-47ef-9c41-f0ce67302a5d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65d8796fc4-9t9t4\" (UID: \"df244f8e-1b94-47ef-9c41-f0ce67302a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4" Sep 09 23:08:49.092967 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:49.092942 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbqr5\" (UniqueName: \"kubernetes.io/projected/df244f8e-1b94-47ef-9c41-f0ce67302a5d-kube-api-access-wbqr5\") pod \"opendatahub-operator-controller-manager-65d8796fc4-9t9t4\" (UID: \"df244f8e-1b94-47ef-9c41-f0ce67302a5d\") " pod="opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4" Sep 09 23:08:49.243616 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:49.243535 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4" Sep 09 23:08:49.539972 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:49.539950 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4"] Sep 09 23:08:49.541660 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:08:49.541635 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf244f8e_1b94_47ef_9c41_f0ce67302a5d.slice/crio-89690f8286e27bf00a1bb95915cfbedab5030ea0622a131f08b37b6972a8a7be WatchSource:0}: Error finding container 89690f8286e27bf00a1bb95915cfbedab5030ea0622a131f08b37b6972a8a7be: Status 404 returned error can't find the container with id 89690f8286e27bf00a1bb95915cfbedab5030ea0622a131f08b37b6972a8a7be Sep 09 23:08:49.906372 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:49.906293 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4" event={"ID":"df244f8e-1b94-47ef-9c41-f0ce67302a5d","Type":"ContainerStarted","Data":"89690f8286e27bf00a1bb95915cfbedab5030ea0622a131f08b37b6972a8a7be"} Sep 09 23:08:52.921219 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:52.921191 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4" event={"ID":"df244f8e-1b94-47ef-9c41-f0ce67302a5d","Type":"ContainerStarted","Data":"a7dbc2042a4087b95488175f91c24c8d067d164de16c5c265d0a40786282d53c"} Sep 09 23:08:53.927540 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:53.927509 2578 generic.go:356] "Generic (PLEG): container finished" podID="df244f8e-1b94-47ef-9c41-f0ce67302a5d" containerID="a7dbc2042a4087b95488175f91c24c8d067d164de16c5c265d0a40786282d53c" exitCode=0 Sep 09 23:08:53.927897 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:53.927573 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4" event={"ID":"df244f8e-1b94-47ef-9c41-f0ce67302a5d","Type":"ContainerDied","Data":"a7dbc2042a4087b95488175f91c24c8d067d164de16c5c265d0a40786282d53c"} Sep 09 23:08:54.933398 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:54.933348 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4" event={"ID":"df244f8e-1b94-47ef-9c41-f0ce67302a5d","Type":"ContainerStarted","Data":"8ed8c19c2a3d8450b5e6113b05bcdd758ca9f2f5b466b8c4245dc7a618f029f6"} Sep 09 23:08:54.933880 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:54.933438 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4" Sep 09 23:08:54.955174 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:08:54.955125 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4" podStartSLOduration=2.395973115 podStartE2EDuration="6.955107778s" totalImagesPullingTime="3.484263944s" totalInitContainerRuntime="1.074870719s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:08:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:08:54.95270401 +0000 UTC m=+595.703288962" watchObservedRunningTime="2026-09-09 23:08:54.955107778 +0000 UTC m=+595.705692706" Sep 09 23:09:05.940484 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:05.940453 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-65d8796fc4-9t9t4" Sep 09 23:09:08.491145 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:08.491103 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7db58bdd4-lkdr8"] Sep 09 23:09:08.493526 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:08.493505 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7db58bdd4-lkdr8" Sep 09 23:09:08.498207 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:08.498188 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 09 23:09:08.498364 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:08.498305 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 09 23:09:08.498364 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:08.498323 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5pswb\"" Sep 09 23:09:08.498857 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:08.498842 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 09 23:09:08.499217 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:08.499166 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 09 23:09:08.513079 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:08.513059 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7db58bdd4-lkdr8"] Sep 09 23:09:08.545721 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:08.545695 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdfxt\" (UniqueName: \"kubernetes.io/projected/5afd01d0-a070-4844-8eff-a6447e22b7d5-kube-api-access-sdfxt\") pod \"kube-auth-proxy-7db58bdd4-lkdr8\" (UID: \"5afd01d0-a070-4844-8eff-a6447e22b7d5\") " pod="openshift-ingress/kube-auth-proxy-7db58bdd4-lkdr8" Sep 09 23:09:08.545865 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:08.545784 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5afd01d0-a070-4844-8eff-a6447e22b7d5-tls-certs\") pod \"kube-auth-proxy-7db58bdd4-lkdr8\" (UID: \"5afd01d0-a070-4844-8eff-a6447e22b7d5\") " pod="openshift-ingress/kube-auth-proxy-7db58bdd4-lkdr8" Sep 09 23:09:08.545865 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:08.545816 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5afd01d0-a070-4844-8eff-a6447e22b7d5-tmp\") pod \"kube-auth-proxy-7db58bdd4-lkdr8\" (UID: \"5afd01d0-a070-4844-8eff-a6447e22b7d5\") " pod="openshift-ingress/kube-auth-proxy-7db58bdd4-lkdr8" Sep 09 23:09:08.647166 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:08.647129 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdfxt\" (UniqueName: \"kubernetes.io/projected/5afd01d0-a070-4844-8eff-a6447e22b7d5-kube-api-access-sdfxt\") pod \"kube-auth-proxy-7db58bdd4-lkdr8\" (UID: \"5afd01d0-a070-4844-8eff-a6447e22b7d5\") " pod="openshift-ingress/kube-auth-proxy-7db58bdd4-lkdr8" Sep 09 23:09:08.647337 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:08.647218 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5afd01d0-a070-4844-8eff-a6447e22b7d5-tls-certs\") pod \"kube-auth-proxy-7db58bdd4-lkdr8\" (UID: \"5afd01d0-a070-4844-8eff-a6447e22b7d5\") " pod="openshift-ingress/kube-auth-proxy-7db58bdd4-lkdr8" Sep 09 23:09:08.647337 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:08.647249 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5afd01d0-a070-4844-8eff-a6447e22b7d5-tmp\") pod \"kube-auth-proxy-7db58bdd4-lkdr8\" (UID: \"5afd01d0-a070-4844-8eff-a6447e22b7d5\") " pod="openshift-ingress/kube-auth-proxy-7db58bdd4-lkdr8" Sep 09 23:09:08.649757 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:08.649735 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5afd01d0-a070-4844-8eff-a6447e22b7d5-tmp\") pod \"kube-auth-proxy-7db58bdd4-lkdr8\" (UID: \"5afd01d0-a070-4844-8eff-a6447e22b7d5\") " pod="openshift-ingress/kube-auth-proxy-7db58bdd4-lkdr8" Sep 09 23:09:08.649881 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:08.649864 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5afd01d0-a070-4844-8eff-a6447e22b7d5-tls-certs\") pod \"kube-auth-proxy-7db58bdd4-lkdr8\" (UID: \"5afd01d0-a070-4844-8eff-a6447e22b7d5\") " pod="openshift-ingress/kube-auth-proxy-7db58bdd4-lkdr8" Sep 09 23:09:08.655826 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:08.655775 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdfxt\" (UniqueName: \"kubernetes.io/projected/5afd01d0-a070-4844-8eff-a6447e22b7d5-kube-api-access-sdfxt\") pod \"kube-auth-proxy-7db58bdd4-lkdr8\" (UID: \"5afd01d0-a070-4844-8eff-a6447e22b7d5\") " pod="openshift-ingress/kube-auth-proxy-7db58bdd4-lkdr8" Sep 09 23:09:08.803665 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:08.803631 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7db58bdd4-lkdr8" Sep 09 23:09:09.079059 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:09.079034 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7db58bdd4-lkdr8"] Sep 09 23:09:09.080330 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:09:09.080302 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5afd01d0_a070_4844_8eff_a6447e22b7d5.slice/crio-3909f7814ba7e4332dde6c3e4079af2012383c3304ed3a428c805b754cc46ce6 WatchSource:0}: Error finding container 3909f7814ba7e4332dde6c3e4079af2012383c3304ed3a428c805b754cc46ce6: Status 404 returned error can't find the container with id 3909f7814ba7e4332dde6c3e4079af2012383c3304ed3a428c805b754cc46ce6 Sep 09 23:09:09.082237 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:09.082224 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 23:09:09.893015 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:09.892925 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7"] Sep 09 23:09:09.895854 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:09.895831 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:09.898450 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:09.898315 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 09 23:09:09.898450 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:09.898315 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 09 23:09:09.898450 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:09.898322 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 09 23:09:09.898664 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:09.898647 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 09 23:09:09.898712 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:09.898695 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gmwhs\"" Sep 09 23:09:09.918202 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:09.916952 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7"] Sep 09 23:09:09.959107 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:09.959071 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q8fc\" (UniqueName: \"kubernetes.io/projected/c37bfb69-930f-4d87-9f6d-aa5818dbf170-kube-api-access-6q8fc\") pod \"istiod-openshift-gateway-7b7995d5cd-bqwn7\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:09.959287 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:09.959122 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c37bfb69-930f-4d87-9f6d-aa5818dbf170-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bqwn7\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:09.959287 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:09.959188 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c37bfb69-930f-4d87-9f6d-aa5818dbf170-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bqwn7\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:09.959287 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:09.959266 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c37bfb69-930f-4d87-9f6d-aa5818dbf170-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bqwn7\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:09.959490 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:09.959305 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37bfb69-930f-4d87-9f6d-aa5818dbf170-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bqwn7\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:09.959490 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:09.959451 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c37bfb69-930f-4d87-9f6d-aa5818dbf170-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bqwn7\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:09.959599 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:09.959496 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37bfb69-930f-4d87-9f6d-aa5818dbf170-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bqwn7\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:10.000600 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:10.000557 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7db58bdd4-lkdr8" event={"ID":"5afd01d0-a070-4844-8eff-a6447e22b7d5","Type":"ContainerStarted","Data":"3909f7814ba7e4332dde6c3e4079af2012383c3304ed3a428c805b754cc46ce6"} Sep 09 23:09:10.060815 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:10.060773 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q8fc\" (UniqueName: \"kubernetes.io/projected/c37bfb69-930f-4d87-9f6d-aa5818dbf170-kube-api-access-6q8fc\") pod \"istiod-openshift-gateway-7b7995d5cd-bqwn7\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:10.060815 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:10.060825 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c37bfb69-930f-4d87-9f6d-aa5818dbf170-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bqwn7\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:10.061084 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:10.060850 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c37bfb69-930f-4d87-9f6d-aa5818dbf170-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bqwn7\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:10.061084 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:10.060878 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c37bfb69-930f-4d87-9f6d-aa5818dbf170-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bqwn7\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:10.061084 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:10.060918 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37bfb69-930f-4d87-9f6d-aa5818dbf170-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bqwn7\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:10.061084 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:10.060987 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c37bfb69-930f-4d87-9f6d-aa5818dbf170-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bqwn7\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:10.061084 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:10.061014 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37bfb69-930f-4d87-9f6d-aa5818dbf170-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bqwn7\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:10.061618 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:10.061561 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c37bfb69-930f-4d87-9f6d-aa5818dbf170-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bqwn7\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:10.063861 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:10.063833 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37bfb69-930f-4d87-9f6d-aa5818dbf170-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bqwn7\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:10.064605 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:10.064556 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c37bfb69-930f-4d87-9f6d-aa5818dbf170-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bqwn7\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:10.064792 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:10.064767 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c37bfb69-930f-4d87-9f6d-aa5818dbf170-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bqwn7\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:10.066004 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:10.065981 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c37bfb69-930f-4d87-9f6d-aa5818dbf170-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bqwn7\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:10.069419 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:10.069370 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37bfb69-930f-4d87-9f6d-aa5818dbf170-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bqwn7\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:10.069817 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:10.069794 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q8fc\" (UniqueName: \"kubernetes.io/projected/c37bfb69-930f-4d87-9f6d-aa5818dbf170-kube-api-access-6q8fc\") pod \"istiod-openshift-gateway-7b7995d5cd-bqwn7\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:10.209797 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:10.209717 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:10.506223 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:10.506111 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7"] Sep 09 23:09:11.783537 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:09:11.783506 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc37bfb69_930f_4d87_9f6d_aa5818dbf170.slice/crio-102989ec4ad58e0e9ff594fca490c8dca56dd2a42a7e7358904e8fcb499343d4 WatchSource:0}: Error finding container 102989ec4ad58e0e9ff594fca490c8dca56dd2a42a7e7358904e8fcb499343d4: Status 404 returned error can't find the container with id 102989ec4ad58e0e9ff594fca490c8dca56dd2a42a7e7358904e8fcb499343d4 Sep 09 23:09:12.013221 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:12.013181 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" event={"ID":"c37bfb69-930f-4d87-9f6d-aa5818dbf170","Type":"ContainerStarted","Data":"102989ec4ad58e0e9ff594fca490c8dca56dd2a42a7e7358904e8fcb499343d4"} Sep 09 23:09:12.014682 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:12.014655 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7db58bdd4-lkdr8" event={"ID":"5afd01d0-a070-4844-8eff-a6447e22b7d5","Type":"ContainerStarted","Data":"f81e10f74d42ffdbad1730412e932f902737fb7c0f837afcaf3cbb3fc89fce51"} Sep 09 23:09:12.030579 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:12.030538 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7db58bdd4-lkdr8" podStartSLOduration=1.293336735 podStartE2EDuration="4.030523365s" totalImagesPullingTime="2.73718663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:09:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:09:12.029245434 +0000 UTC m=+612.779830410" watchObservedRunningTime="2026-09-09 23:09:12.030523365 +0000 UTC m=+612.781108323" Sep 09 23:09:14.563794 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:14.563751 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 09 23:09:14.564087 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:14.563820 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 09 23:09:15.032416 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:15.032367 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" event={"ID":"c37bfb69-930f-4d87-9f6d-aa5818dbf170","Type":"ContainerStarted","Data":"9ede8b865b7e2577d5d5697d2fe039fcd1f12391d4531cee7872385fe1ce7b8c"} Sep 09 23:09:15.032616 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:15.032597 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:15.034173 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:15.034151 2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-bqwn7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 09 23:09:15.034280 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:15.034200 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" podUID="c37bfb69-930f-4d87-9f6d-aa5818dbf170" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 09 23:09:15.051653 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:15.051606 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" podStartSLOduration=3.2734260600000002 podStartE2EDuration="6.051592813s" totalImagesPullingTime="2.778166753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:09:15.050942325 +0000 UTC m=+615.801527251" watchObservedRunningTime="2026-09-09 23:09:15.051592813 +0000 UTC m=+615.802177740" Sep 09 23:09:16.037272 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:16.037245 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:19.677131 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:19.677094 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-rb4ng"] Sep 09 23:09:19.681844 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:19.681824 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-rb4ng" Sep 09 23:09:19.684253 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:19.684227 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 09 23:09:19.684253 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:19.684245 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-bg5mm\"" Sep 09 23:09:19.691296 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:19.691274 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-rb4ng"] Sep 09 23:09:19.747784 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:19.747753 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b353b9-a213-40d3-993f-ac949389e2f4-tls-certs\") pod \"model-serving-api-54d89bd5fc-rb4ng\" (UID: \"94b353b9-a213-40d3-993f-ac949389e2f4\") " pod="opendatahub/model-serving-api-54d89bd5fc-rb4ng" Sep 09 23:09:19.747950 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:19.747801 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9c5r\" (UniqueName: \"kubernetes.io/projected/94b353b9-a213-40d3-993f-ac949389e2f4-kube-api-access-r9c5r\") pod \"model-serving-api-54d89bd5fc-rb4ng\" (UID: \"94b353b9-a213-40d3-993f-ac949389e2f4\") " pod="opendatahub/model-serving-api-54d89bd5fc-rb4ng" Sep 09 23:09:19.848366 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:19.848332 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b353b9-a213-40d3-993f-ac949389e2f4-tls-certs\") pod \"model-serving-api-54d89bd5fc-rb4ng\" (UID: \"94b353b9-a213-40d3-993f-ac949389e2f4\") " pod="opendatahub/model-serving-api-54d89bd5fc-rb4ng" Sep 09 23:09:19.848555 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:19.848456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9c5r\" (UniqueName: \"kubernetes.io/projected/94b353b9-a213-40d3-993f-ac949389e2f4-kube-api-access-r9c5r\") pod \"model-serving-api-54d89bd5fc-rb4ng\" (UID: \"94b353b9-a213-40d3-993f-ac949389e2f4\") " pod="opendatahub/model-serving-api-54d89bd5fc-rb4ng" Sep 09 23:09:19.848555 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:09:19.848487 2578 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 09 23:09:19.848687 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:09:19.848574 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94b353b9-a213-40d3-993f-ac949389e2f4-tls-certs podName:94b353b9-a213-40d3-993f-ac949389e2f4 nodeName:}" failed. No retries permitted until 2026-09-09 23:09:20.348553097 +0000 UTC m=+621.099138022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/94b353b9-a213-40d3-993f-ac949389e2f4-tls-certs") pod "model-serving-api-54d89bd5fc-rb4ng" (UID: "94b353b9-a213-40d3-993f-ac949389e2f4") : secret "model-serving-api-tls" not found Sep 09 23:09:19.857942 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:19.857914 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9c5r\" (UniqueName: \"kubernetes.io/projected/94b353b9-a213-40d3-993f-ac949389e2f4-kube-api-access-r9c5r\") pod \"model-serving-api-54d89bd5fc-rb4ng\" (UID: \"94b353b9-a213-40d3-993f-ac949389e2f4\") " pod="opendatahub/model-serving-api-54d89bd5fc-rb4ng" Sep 09 23:09:20.354524 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:20.354476 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b353b9-a213-40d3-993f-ac949389e2f4-tls-certs\") pod \"model-serving-api-54d89bd5fc-rb4ng\" (UID: \"94b353b9-a213-40d3-993f-ac949389e2f4\") " pod="opendatahub/model-serving-api-54d89bd5fc-rb4ng" Sep 09 23:09:20.357170 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:20.357131 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b353b9-a213-40d3-993f-ac949389e2f4-tls-certs\") pod \"model-serving-api-54d89bd5fc-rb4ng\" (UID: \"94b353b9-a213-40d3-993f-ac949389e2f4\") " pod="opendatahub/model-serving-api-54d89bd5fc-rb4ng" Sep 09 23:09:20.592449 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:20.592361 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-rb4ng" Sep 09 23:09:20.862151 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:20.862033 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-rb4ng"] Sep 09 23:09:20.865116 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:09:20.865090 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b353b9_a213_40d3_993f_ac949389e2f4.slice/crio-dae4d1c4f1358cc9b7b50ea41017772f0f296c718b0af5cf13cd26696f17a79c WatchSource:0}: Error finding container dae4d1c4f1358cc9b7b50ea41017772f0f296c718b0af5cf13cd26696f17a79c: Status 404 returned error can't find the container with id dae4d1c4f1358cc9b7b50ea41017772f0f296c718b0af5cf13cd26696f17a79c Sep 09 23:09:21.066124 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:21.066086 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-rb4ng" event={"ID":"94b353b9-a213-40d3-993f-ac949389e2f4","Type":"ContainerStarted","Data":"dae4d1c4f1358cc9b7b50ea41017772f0f296c718b0af5cf13cd26696f17a79c"} Sep 09 23:09:23.079920 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:23.079878 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-rb4ng" event={"ID":"94b353b9-a213-40d3-993f-ac949389e2f4","Type":"ContainerStarted","Data":"9ed07793e355ffd7ebfcd0111138d72dfd850a30d970087cab596c8f6c838a6d"} Sep 09 23:09:23.080297 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:23.079978 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-rb4ng" Sep 09 23:09:23.097421 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:23.097354 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-rb4ng" podStartSLOduration=2.514216935 podStartE2EDuration="4.097333877s" totalImagesPullingTime="1.583116942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:09:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:09:23.094823272 +0000 UTC m=+623.845408212" watchObservedRunningTime="2026-09-09 23:09:23.097333877 +0000 UTC m=+623.847918805" Sep 09 23:09:28.872671 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:28.872632 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4sss6"] Sep 09 23:09:28.875950 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:28.875927 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4sss6" Sep 09 23:09:28.883567 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:28.883544 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 09 23:09:28.883702 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:28.883631 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-5n9gr\"" Sep 09 23:09:28.895015 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:28.894986 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wzvqr"] Sep 09 23:09:28.898191 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:28.898170 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wzvqr" Sep 09 23:09:28.900862 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:28.900835 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4sss6"] Sep 09 23:09:28.900973 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:28.900958 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 09 23:09:28.901043 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:28.900978 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-6rp6t\"" Sep 09 23:09:28.916812 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:28.916783 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wzvqr"] Sep 09 23:09:29.034331 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:29.034283 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqp7w\" (UniqueName: \"kubernetes.io/projected/3828e5a3-af2d-4422-8fea-b9fd6fafd1ef-kube-api-access-hqp7w\") pod \"llmisvc-controller-manager-777c6bd7d5-wzvqr\" (UID: \"3828e5a3-af2d-4422-8fea-b9fd6fafd1ef\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wzvqr" Sep 09 23:09:29.034509 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:29.034345 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/329724e3-d1fb-46c9-b267-061c965cede0-cert\") pod \"kserve-controller-manager-d6f4b65b8-4sss6\" (UID: \"329724e3-d1fb-46c9-b267-061c965cede0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4sss6" Sep 09 23:09:29.034509 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:29.034420 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp689\" (UniqueName: \"kubernetes.io/projected/329724e3-d1fb-46c9-b267-061c965cede0-kube-api-access-qp689\") pod \"kserve-controller-manager-d6f4b65b8-4sss6\" (UID: \"329724e3-d1fb-46c9-b267-061c965cede0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4sss6" Sep 09 23:09:29.034509 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:29.034457 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3828e5a3-af2d-4422-8fea-b9fd6fafd1ef-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wzvqr\" (UID: \"3828e5a3-af2d-4422-8fea-b9fd6fafd1ef\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wzvqr" Sep 09 23:09:29.135541 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:29.135438 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqp7w\" (UniqueName: \"kubernetes.io/projected/3828e5a3-af2d-4422-8fea-b9fd6fafd1ef-kube-api-access-hqp7w\") pod \"llmisvc-controller-manager-777c6bd7d5-wzvqr\" (UID: \"3828e5a3-af2d-4422-8fea-b9fd6fafd1ef\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wzvqr" Sep 09 23:09:29.135541 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:29.135497 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/329724e3-d1fb-46c9-b267-061c965cede0-cert\") pod \"kserve-controller-manager-d6f4b65b8-4sss6\" (UID: \"329724e3-d1fb-46c9-b267-061c965cede0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4sss6" Sep 09 23:09:29.135769 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:29.135568 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp689\" (UniqueName: \"kubernetes.io/projected/329724e3-d1fb-46c9-b267-061c965cede0-kube-api-access-qp689\") pod \"kserve-controller-manager-d6f4b65b8-4sss6\" (UID: \"329724e3-d1fb-46c9-b267-061c965cede0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4sss6" Sep 09 23:09:29.135769 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:29.135598 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3828e5a3-af2d-4422-8fea-b9fd6fafd1ef-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wzvqr\" (UID: \"3828e5a3-af2d-4422-8fea-b9fd6fafd1ef\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wzvqr" Sep 09 23:09:29.135879 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:09:29.135784 2578 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 09 23:09:29.135879 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:09:29.135849 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3828e5a3-af2d-4422-8fea-b9fd6fafd1ef-cert podName:3828e5a3-af2d-4422-8fea-b9fd6fafd1ef nodeName:}" failed. No retries permitted until 2026-09-09 23:09:29.635827241 +0000 UTC m=+630.386412150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3828e5a3-af2d-4422-8fea-b9fd6fafd1ef-cert") pod "llmisvc-controller-manager-777c6bd7d5-wzvqr" (UID: "3828e5a3-af2d-4422-8fea-b9fd6fafd1ef") : secret "llmisvc-webhook-server-cert" not found Sep 09 23:09:29.136415 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:09:29.136373 2578 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 09 23:09:29.136480 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:09:29.136450 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/329724e3-d1fb-46c9-b267-061c965cede0-cert podName:329724e3-d1fb-46c9-b267-061c965cede0 nodeName:}" failed. No retries permitted until 2026-09-09 23:09:29.636434437 +0000 UTC m=+630.387019343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/329724e3-d1fb-46c9-b267-061c965cede0-cert") pod "kserve-controller-manager-d6f4b65b8-4sss6" (UID: "329724e3-d1fb-46c9-b267-061c965cede0") : secret "kserve-webhook-server-cert" not found Sep 09 23:09:29.164272 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:29.157699 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqp7w\" (UniqueName: \"kubernetes.io/projected/3828e5a3-af2d-4422-8fea-b9fd6fafd1ef-kube-api-access-hqp7w\") pod \"llmisvc-controller-manager-777c6bd7d5-wzvqr\" (UID: \"3828e5a3-af2d-4422-8fea-b9fd6fafd1ef\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wzvqr" Sep 09 23:09:29.164272 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:29.159846 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp689\" (UniqueName: \"kubernetes.io/projected/329724e3-d1fb-46c9-b267-061c965cede0-kube-api-access-qp689\") pod \"kserve-controller-manager-d6f4b65b8-4sss6\" (UID: \"329724e3-d1fb-46c9-b267-061c965cede0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4sss6" Sep 09 23:09:29.640914 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:29.640879 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/329724e3-d1fb-46c9-b267-061c965cede0-cert\") pod \"kserve-controller-manager-d6f4b65b8-4sss6\" (UID: \"329724e3-d1fb-46c9-b267-061c965cede0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4sss6" Sep 09 23:09:29.641094 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:29.640951 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3828e5a3-af2d-4422-8fea-b9fd6fafd1ef-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wzvqr\" (UID: \"3828e5a3-af2d-4422-8fea-b9fd6fafd1ef\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wzvqr" Sep 09 23:09:29.643486 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:29.643459 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3828e5a3-af2d-4422-8fea-b9fd6fafd1ef-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wzvqr\" (UID: \"3828e5a3-af2d-4422-8fea-b9fd6fafd1ef\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wzvqr" Sep 09 23:09:29.643616 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:29.643595 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/329724e3-d1fb-46c9-b267-061c965cede0-cert\") pod \"kserve-controller-manager-d6f4b65b8-4sss6\" (UID: \"329724e3-d1fb-46c9-b267-061c965cede0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4sss6" Sep 09 23:09:29.787267 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:29.787224 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4sss6" Sep 09 23:09:29.810004 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:29.809973 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wzvqr" Sep 09 23:09:30.092620 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.092594 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4sss6"] Sep 09 23:09:30.094271 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:09:30.094247 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329724e3_d1fb_46c9_b267_061c965cede0.slice/crio-663417b08eac3ccd6b0248b11b81925a5025e482594e49608a956f6e3388ed42 WatchSource:0}: Error finding container 663417b08eac3ccd6b0248b11b81925a5025e482594e49608a956f6e3388ed42: Status 404 returned error can't find the container with id 663417b08eac3ccd6b0248b11b81925a5025e482594e49608a956f6e3388ed42 Sep 09 23:09:30.110233 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.107671 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wzvqr"] Sep 09 23:09:30.111651 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:09:30.111624 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3828e5a3_af2d_4422_8fea_b9fd6fafd1ef.slice/crio-c021d736a1eb6a305d20ba983b1141074f31f9a102bf9906afc563099587d677 WatchSource:0}: Error finding container c021d736a1eb6a305d20ba983b1141074f31f9a102bf9906afc563099587d677: Status 404 returned error can't find the container with id c021d736a1eb6a305d20ba983b1141074f31f9a102bf9906afc563099587d677 Sep 09 23:09:30.115999 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.115975 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4sss6" event={"ID":"329724e3-d1fb-46c9-b267-061c965cede0","Type":"ContainerStarted","Data":"663417b08eac3ccd6b0248b11b81925a5025e482594e49608a956f6e3388ed42"} Sep 09 23:09:30.261428 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.261396 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7"] Sep 09 23:09:30.261596 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.261574 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" podUID="c37bfb69-930f-4d87-9f6d-aa5818dbf170" containerName="discovery" containerID="cri-o://9ede8b865b7e2577d5d5697d2fe039fcd1f12391d4531cee7872385fe1ce7b8c" gracePeriod=30 Sep 09 23:09:30.500543 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.500520 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:30.650364 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.650286 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6q8fc\" (UniqueName: \"kubernetes.io/projected/c37bfb69-930f-4d87-9f6d-aa5818dbf170-kube-api-access-6q8fc\") pod \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " Sep 09 23:09:30.650364 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.650319 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37bfb69-930f-4d87-9f6d-aa5818dbf170-local-certs\") pod \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " Sep 09 23:09:30.650606 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.650367 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c37bfb69-930f-4d87-9f6d-aa5818dbf170-cacerts\") pod \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " Sep 09 23:09:30.650606 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.650416 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c37bfb69-930f-4d87-9f6d-aa5818dbf170-istio-kubeconfig\") pod \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " Sep 09 23:09:30.650606 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.650451 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c37bfb69-930f-4d87-9f6d-aa5818dbf170-istio-csr-ca-configmap\") pod \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " Sep 09 23:09:30.650606 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.650499 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c37bfb69-930f-4d87-9f6d-aa5818dbf170-istio-csr-dns-cert\") pod \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " Sep 09 23:09:30.650606 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.650522 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37bfb69-930f-4d87-9f6d-aa5818dbf170-istio-token\") pod \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\" (UID: \"c37bfb69-930f-4d87-9f6d-aa5818dbf170\") " Sep 09 23:09:30.651303 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.651257 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c37bfb69-930f-4d87-9f6d-aa5818dbf170-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "c37bfb69-930f-4d87-9f6d-aa5818dbf170" (UID: "c37bfb69-930f-4d87-9f6d-aa5818dbf170"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:09:30.653467 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.653438 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c37bfb69-930f-4d87-9f6d-aa5818dbf170-cacerts" (OuterVolumeSpecName: "cacerts") pod "c37bfb69-930f-4d87-9f6d-aa5818dbf170" (UID: "c37bfb69-930f-4d87-9f6d-aa5818dbf170"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:09:30.653597 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.653470 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c37bfb69-930f-4d87-9f6d-aa5818dbf170-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "c37bfb69-930f-4d87-9f6d-aa5818dbf170" (UID: "c37bfb69-930f-4d87-9f6d-aa5818dbf170"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:09:30.653675 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.653612 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37bfb69-930f-4d87-9f6d-aa5818dbf170-local-certs" (OuterVolumeSpecName: "local-certs") pod "c37bfb69-930f-4d87-9f6d-aa5818dbf170" (UID: "c37bfb69-930f-4d87-9f6d-aa5818dbf170"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:09:30.653675 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.653611 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c37bfb69-930f-4d87-9f6d-aa5818dbf170-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "c37bfb69-930f-4d87-9f6d-aa5818dbf170" (UID: "c37bfb69-930f-4d87-9f6d-aa5818dbf170"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:09:30.653756 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.653714 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c37bfb69-930f-4d87-9f6d-aa5818dbf170-kube-api-access-6q8fc" (OuterVolumeSpecName: "kube-api-access-6q8fc") pod "c37bfb69-930f-4d87-9f6d-aa5818dbf170" (UID: "c37bfb69-930f-4d87-9f6d-aa5818dbf170"). InnerVolumeSpecName "kube-api-access-6q8fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:09:30.653756 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.653737 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c37bfb69-930f-4d87-9f6d-aa5818dbf170-istio-token" (OuterVolumeSpecName: "istio-token") pod "c37bfb69-930f-4d87-9f6d-aa5818dbf170" (UID: "c37bfb69-930f-4d87-9f6d-aa5818dbf170"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:09:30.751750 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.751716 2578 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c37bfb69-930f-4d87-9f6d-aa5818dbf170-cacerts\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:09:30.751750 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.751750 2578 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c37bfb69-930f-4d87-9f6d-aa5818dbf170-istio-kubeconfig\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:09:30.751982 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.751766 2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c37bfb69-930f-4d87-9f6d-aa5818dbf170-istio-csr-ca-configmap\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:09:30.751982 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.751781 2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c37bfb69-930f-4d87-9f6d-aa5818dbf170-istio-csr-dns-cert\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:09:30.751982 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.751794 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37bfb69-930f-4d87-9f6d-aa5818dbf170-istio-token\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:09:30.751982 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.751807 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q8fc\" (UniqueName: \"kubernetes.io/projected/c37bfb69-930f-4d87-9f6d-aa5818dbf170-kube-api-access-6q8fc\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:09:30.751982 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:30.751819 2578 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37bfb69-930f-4d87-9f6d-aa5818dbf170-local-certs\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:09:31.125714 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:31.125676 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wzvqr" event={"ID":"3828e5a3-af2d-4422-8fea-b9fd6fafd1ef","Type":"ContainerStarted","Data":"c021d736a1eb6a305d20ba983b1141074f31f9a102bf9906afc563099587d677"} Sep 09 23:09:31.127328 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:31.127295 2578 generic.go:356] "Generic (PLEG): container finished" podID="c37bfb69-930f-4d87-9f6d-aa5818dbf170" containerID="9ede8b865b7e2577d5d5697d2fe039fcd1f12391d4531cee7872385fe1ce7b8c" exitCode=0 Sep 09 23:09:31.127472 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:31.127340 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" event={"ID":"c37bfb69-930f-4d87-9f6d-aa5818dbf170","Type":"ContainerDied","Data":"9ede8b865b7e2577d5d5697d2fe039fcd1f12391d4531cee7872385fe1ce7b8c"} Sep 09 23:09:31.127472 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:31.127351 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" Sep 09 23:09:31.127472 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:31.127413 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7" event={"ID":"c37bfb69-930f-4d87-9f6d-aa5818dbf170","Type":"ContainerDied","Data":"102989ec4ad58e0e9ff594fca490c8dca56dd2a42a7e7358904e8fcb499343d4"} Sep 09 23:09:31.127472 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:31.127441 2578 scope.go:117] "RemoveContainer" containerID="9ede8b865b7e2577d5d5697d2fe039fcd1f12391d4531cee7872385fe1ce7b8c" Sep 09 23:09:31.153567 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:31.153536 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7"] Sep 09 23:09:31.157835 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:31.157811 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bqwn7"] Sep 09 23:09:31.279827 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:31.279797 2578 scope.go:117] "RemoveContainer" containerID="9ede8b865b7e2577d5d5697d2fe039fcd1f12391d4531cee7872385fe1ce7b8c" Sep 09 23:09:31.280215 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:09:31.280192 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ede8b865b7e2577d5d5697d2fe039fcd1f12391d4531cee7872385fe1ce7b8c\": container with ID starting with 9ede8b865b7e2577d5d5697d2fe039fcd1f12391d4531cee7872385fe1ce7b8c not found: ID does not exist" containerID="9ede8b865b7e2577d5d5697d2fe039fcd1f12391d4531cee7872385fe1ce7b8c" Sep 09 23:09:31.280315 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:31.280225 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ede8b865b7e2577d5d5697d2fe039fcd1f12391d4531cee7872385fe1ce7b8c"} err="failed to get container status \"9ede8b865b7e2577d5d5697d2fe039fcd1f12391d4531cee7872385fe1ce7b8c\": rpc error: code = NotFound desc = could not find container \"9ede8b865b7e2577d5d5697d2fe039fcd1f12391d4531cee7872385fe1ce7b8c\": container with ID starting with 9ede8b865b7e2577d5d5697d2fe039fcd1f12391d4531cee7872385fe1ce7b8c not found: ID does not exist" Sep 09 23:09:31.852301 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:31.852271 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c37bfb69-930f-4d87-9f6d-aa5818dbf170" path="/var/lib/kubelet/pods/c37bfb69-930f-4d87-9f6d-aa5818dbf170/volumes" Sep 09 23:09:33.142037 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:33.141997 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4sss6" event={"ID":"329724e3-d1fb-46c9-b267-061c965cede0","Type":"ContainerStarted","Data":"345a724a230bf03e3ac75a308271b562025d2ba27362cd8139e718584f98a897"} Sep 09 23:09:33.142514 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:33.142148 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4sss6" Sep 09 23:09:33.143719 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:33.143693 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wzvqr" event={"ID":"3828e5a3-af2d-4422-8fea-b9fd6fafd1ef","Type":"ContainerStarted","Data":"8c267664280c0d3db32526422bc1f57a92af144d1979befc20d99b5a3f93505f"} Sep 09 23:09:33.143834 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:33.143758 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wzvqr" Sep 09 23:09:33.159793 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:33.159750 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4sss6" podStartSLOduration=2.2848465940000002 podStartE2EDuration="5.159738233s" totalImagesPullingTime="2.874891639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:09:33.158225887 +0000 UTC m=+633.908810825" watchObservedRunningTime="2026-09-09 23:09:33.159738233 +0000 UTC m=+633.910323160" Sep 09 23:09:33.186425 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:33.186365 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wzvqr" podStartSLOduration=2.326176682 podStartE2EDuration="5.186352186s" totalImagesPullingTime="2.860175504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:09:33.18349821 +0000 UTC m=+633.934083156" watchObservedRunningTime="2026-09-09 23:09:33.186352186 +0000 UTC m=+633.936937119" Sep 09 23:09:34.090014 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:09:34.089990 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-rb4ng" Sep 09 23:10:04.151262 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:04.151227 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wzvqr" Sep 09 23:10:04.154210 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:04.154192 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4sss6" Sep 09 23:10:37.538455 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:37.538417 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sp7hl"] Sep 09 23:10:37.538994 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:37.538810 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c37bfb69-930f-4d87-9f6d-aa5818dbf170" containerName="discovery" Sep 09 23:10:37.538994 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:37.538822 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c37bfb69-930f-4d87-9f6d-aa5818dbf170" containerName="discovery" Sep 09 23:10:37.538994 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:37.538888 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c37bfb69-930f-4d87-9f6d-aa5818dbf170" containerName="discovery" Sep 09 23:10:37.540918 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:37.540901 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sp7hl" Sep 09 23:10:37.543371 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:37.543351 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-2pps9\"" Sep 09 23:10:37.543579 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:37.543561 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 09 23:10:37.543673 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:37.543657 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 09 23:10:37.545980 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:37.545962 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 09 23:10:37.559405 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:37.559358 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sp7hl"] Sep 09 23:10:37.624845 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:37.624816 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jzbj\" (UniqueName: \"kubernetes.io/projected/52d176e8-bec3-4248-a2eb-acb48c50e1b0-kube-api-access-4jzbj\") pod \"dns-operator-controller-manager-7b84cd7c9c-sp7hl\" (UID: \"52d176e8-bec3-4248-a2eb-acb48c50e1b0\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sp7hl" Sep 09 23:10:37.725471 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:37.725439 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jzbj\" (UniqueName: \"kubernetes.io/projected/52d176e8-bec3-4248-a2eb-acb48c50e1b0-kube-api-access-4jzbj\") pod \"dns-operator-controller-manager-7b84cd7c9c-sp7hl\" (UID: \"52d176e8-bec3-4248-a2eb-acb48c50e1b0\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sp7hl" Sep 09 23:10:37.751436 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:37.751398 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jzbj\" (UniqueName: \"kubernetes.io/projected/52d176e8-bec3-4248-a2eb-acb48c50e1b0-kube-api-access-4jzbj\") pod \"dns-operator-controller-manager-7b84cd7c9c-sp7hl\" (UID: \"52d176e8-bec3-4248-a2eb-acb48c50e1b0\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sp7hl" Sep 09 23:10:37.851288 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:37.851211 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sp7hl" Sep 09 23:10:38.141645 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:38.141604 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sp7hl"] Sep 09 23:10:38.143426 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:10:38.143367 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d176e8_bec3_4248_a2eb_acb48c50e1b0.slice/crio-68dd4dbcd7f6da910b2f01836f238258d9a5b2c2e3325ca97638bcbc09f08057 WatchSource:0}: Error finding container 68dd4dbcd7f6da910b2f01836f238258d9a5b2c2e3325ca97638bcbc09f08057: Status 404 returned error can't find the container with id 68dd4dbcd7f6da910b2f01836f238258d9a5b2c2e3325ca97638bcbc09f08057 Sep 09 23:10:38.439913 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:38.439821 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sp7hl" event={"ID":"52d176e8-bec3-4248-a2eb-acb48c50e1b0","Type":"ContainerStarted","Data":"68dd4dbcd7f6da910b2f01836f238258d9a5b2c2e3325ca97638bcbc09f08057"} Sep 09 23:10:40.165099 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:40.165064 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-qxxdk"] Sep 09 23:10:40.172368 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:40.172345 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-qxxdk" Sep 09 23:10:40.174879 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:40.174859 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-8vpdr\"" Sep 09 23:10:40.180771 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:40.180745 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-qxxdk"] Sep 09 23:10:40.247069 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:40.247038 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48hwd\" (UniqueName: \"kubernetes.io/projected/73149d7d-077d-43da-ab09-05b93da15865-kube-api-access-48hwd\") pod \"authorino-operator-76b478bd78-qxxdk\" (UID: \"73149d7d-077d-43da-ab09-05b93da15865\") " pod="kuadrant-system/authorino-operator-76b478bd78-qxxdk" Sep 09 23:10:40.347955 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:40.347922 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48hwd\" (UniqueName: \"kubernetes.io/projected/73149d7d-077d-43da-ab09-05b93da15865-kube-api-access-48hwd\") pod \"authorino-operator-76b478bd78-qxxdk\" (UID: \"73149d7d-077d-43da-ab09-05b93da15865\") " pod="kuadrant-system/authorino-operator-76b478bd78-qxxdk" Sep 09 23:10:40.356981 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:40.356955 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48hwd\" (UniqueName: \"kubernetes.io/projected/73149d7d-077d-43da-ab09-05b93da15865-kube-api-access-48hwd\") pod \"authorino-operator-76b478bd78-qxxdk\" (UID: \"73149d7d-077d-43da-ab09-05b93da15865\") " pod="kuadrant-system/authorino-operator-76b478bd78-qxxdk" Sep 09 23:10:40.485914 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:40.485831 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-qxxdk" Sep 09 23:10:40.993722 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:40.993697 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-qxxdk"] Sep 09 23:10:40.995598 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:10:40.995573 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73149d7d_077d_43da_ab09_05b93da15865.slice/crio-95bf93ab5d14bbdd8c3a02c26ad70a96882b36a2108357d151551aa7266cd2e2 WatchSource:0}: Error finding container 95bf93ab5d14bbdd8c3a02c26ad70a96882b36a2108357d151551aa7266cd2e2: Status 404 returned error can't find the container with id 95bf93ab5d14bbdd8c3a02c26ad70a96882b36a2108357d151551aa7266cd2e2 Sep 09 23:10:41.460922 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:41.460888 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sp7hl" event={"ID":"52d176e8-bec3-4248-a2eb-acb48c50e1b0","Type":"ContainerStarted","Data":"9f6784a23158cd7931b9345efd9587f04254f0135ba3e4cae1136718525154d1"} Sep 09 23:10:41.461420 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:41.461019 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sp7hl" Sep 09 23:10:41.462117 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:41.462098 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-qxxdk" event={"ID":"73149d7d-077d-43da-ab09-05b93da15865","Type":"ContainerStarted","Data":"95bf93ab5d14bbdd8c3a02c26ad70a96882b36a2108357d151551aa7266cd2e2"} Sep 09 23:10:41.495141 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:41.495100 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sp7hl" podStartSLOduration=1.8699495339999999 podStartE2EDuration="4.495086959s" totalImagesPullingTime="2.625137425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:10:41.492995636 +0000 UTC m=+702.243580583" watchObservedRunningTime="2026-09-09 23:10:41.495086959 +0000 UTC m=+702.245671887" Sep 09 23:10:43.476190 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:43.476154 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-qxxdk" event={"ID":"73149d7d-077d-43da-ab09-05b93da15865","Type":"ContainerStarted","Data":"bc22d78266fa542c0be4cd3a91c4d2a8a2dae4f3b16100b22c205d6f3790953c"} Sep 09 23:10:43.476631 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:43.476260 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-qxxdk" Sep 09 23:10:43.500040 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:43.499992 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-qxxdk" podStartSLOduration=1.6826277950000001 podStartE2EDuration="3.499978673s" totalImagesPullingTime="1.817350878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:10:43.498304261 +0000 UTC m=+704.248889211" watchObservedRunningTime="2026-09-09 23:10:43.499978673 +0000 UTC m=+704.250563600" Sep 09 23:10:44.609666 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:44.609623 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv"] Sep 09 23:10:44.613305 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:44.613289 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv" Sep 09 23:10:44.615994 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:44.615974 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-7wz8n\"" Sep 09 23:10:44.632663 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:44.632642 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv"] Sep 09 23:10:44.684985 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:44.684952 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0482c1e4-668b-4546-839c-d5c99ceb65a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hm6xv\" (UID: \"0482c1e4-668b-4546-839c-d5c99ceb65a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv" Sep 09 23:10:44.685121 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:44.684999 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x5sc\" (UniqueName: \"kubernetes.io/projected/0482c1e4-668b-4546-839c-d5c99ceb65a1-kube-api-access-4x5sc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hm6xv\" (UID: \"0482c1e4-668b-4546-839c-d5c99ceb65a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv" Sep 09 23:10:44.785791 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:44.785765 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x5sc\" (UniqueName: \"kubernetes.io/projected/0482c1e4-668b-4546-839c-d5c99ceb65a1-kube-api-access-4x5sc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hm6xv\" (UID: \"0482c1e4-668b-4546-839c-d5c99ceb65a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv" Sep 09 23:10:44.785891 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:44.785824 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0482c1e4-668b-4546-839c-d5c99ceb65a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hm6xv\" (UID: \"0482c1e4-668b-4546-839c-d5c99ceb65a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv" Sep 09 23:10:44.786115 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:44.786101 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0482c1e4-668b-4546-839c-d5c99ceb65a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hm6xv\" (UID: \"0482c1e4-668b-4546-839c-d5c99ceb65a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv" Sep 09 23:10:44.794416 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:44.794372 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x5sc\" (UniqueName: \"kubernetes.io/projected/0482c1e4-668b-4546-839c-d5c99ceb65a1-kube-api-access-4x5sc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hm6xv\" (UID: \"0482c1e4-668b-4546-839c-d5c99ceb65a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv" Sep 09 23:10:44.923500 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:44.923424 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv" Sep 09 23:10:45.251434 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:45.251410 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv"] Sep 09 23:10:45.253846 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:10:45.253813 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0482c1e4_668b_4546_839c_d5c99ceb65a1.slice/crio-5315647b3933271dfb8dd10df3b2553e31b2bcec6df8debed577f36ceea6cbf7 WatchSource:0}: Error finding container 5315647b3933271dfb8dd10df3b2553e31b2bcec6df8debed577f36ceea6cbf7: Status 404 returned error can't find the container with id 5315647b3933271dfb8dd10df3b2553e31b2bcec6df8debed577f36ceea6cbf7 Sep 09 23:10:45.486687 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:45.486650 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv" event={"ID":"0482c1e4-668b-4546-839c-d5c99ceb65a1","Type":"ContainerStarted","Data":"5315647b3933271dfb8dd10df3b2553e31b2bcec6df8debed577f36ceea6cbf7"} Sep 09 23:10:46.605898 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:46.605861 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rn2fg"] Sep 09 23:10:46.609861 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:46.609833 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rn2fg" Sep 09 23:10:46.612646 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:46.612618 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-bz287\"" Sep 09 23:10:46.621690 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:46.621666 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rn2fg"] Sep 09 23:10:46.703727 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:46.703691 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj2jb\" (UniqueName: \"kubernetes.io/projected/189277a5-595d-4c84-bf48-81247b31983b-kube-api-access-bj2jb\") pod \"limitador-operator-controller-manager-74495c69f8-rn2fg\" (UID: \"189277a5-595d-4c84-bf48-81247b31983b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rn2fg" Sep 09 23:10:46.804951 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:46.804900 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj2jb\" (UniqueName: \"kubernetes.io/projected/189277a5-595d-4c84-bf48-81247b31983b-kube-api-access-bj2jb\") pod \"limitador-operator-controller-manager-74495c69f8-rn2fg\" (UID: \"189277a5-595d-4c84-bf48-81247b31983b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rn2fg" Sep 09 23:10:46.815071 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:46.815038 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj2jb\" (UniqueName: \"kubernetes.io/projected/189277a5-595d-4c84-bf48-81247b31983b-kube-api-access-bj2jb\") pod \"limitador-operator-controller-manager-74495c69f8-rn2fg\" (UID: \"189277a5-595d-4c84-bf48-81247b31983b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rn2fg" Sep 09 23:10:46.923293 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:46.923211 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rn2fg" Sep 09 23:10:47.217693 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:47.217554 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rn2fg"] Sep 09 23:10:47.219923 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:10:47.219894 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189277a5_595d_4c84_bf48_81247b31983b.slice/crio-7e45969171c1277f0e255aa46f76fe3c7b663e1400d60f14a00cc2d1c818c33a WatchSource:0}: Error finding container 7e45969171c1277f0e255aa46f76fe3c7b663e1400d60f14a00cc2d1c818c33a: Status 404 returned error can't find the container with id 7e45969171c1277f0e255aa46f76fe3c7b663e1400d60f14a00cc2d1c818c33a Sep 09 23:10:47.499989 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:47.499909 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rn2fg" event={"ID":"189277a5-595d-4c84-bf48-81247b31983b","Type":"ContainerStarted","Data":"7e45969171c1277f0e255aa46f76fe3c7b663e1400d60f14a00cc2d1c818c33a"} Sep 09 23:10:49.513346 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:49.513286 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv" event={"ID":"0482c1e4-668b-4546-839c-d5c99ceb65a1","Type":"ContainerStarted","Data":"4b0dc7d830adc2bfac4f6fb3c8d99ab361b0bca3f2da34455d92fdf5e2d82b8e"} Sep 09 23:10:49.513796 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:49.513566 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv" Sep 09 23:10:49.540469 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:49.540198 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv" podStartSLOduration=1.478113072 podStartE2EDuration="5.540179178s" totalImagesPullingTime="4.062066106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:10:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:10:49.538650706 +0000 UTC m=+710.289235644" watchObservedRunningTime="2026-09-09 23:10:49.540179178 +0000 UTC m=+710.290764106" Sep 09 23:10:50.520705 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:50.520672 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rn2fg" event={"ID":"189277a5-595d-4c84-bf48-81247b31983b","Type":"ContainerStarted","Data":"42dc23aaf80c5fa4f8ce7134636a1b851d3cc285f14dd84bc759f6570ed5ee7e"} Sep 09 23:10:50.540268 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:50.540189 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rn2fg" podStartSLOduration=1.557636371 podStartE2EDuration="4.540172896s" totalImagesPullingTime="2.982536525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:10:50.539001997 +0000 UTC m=+711.289586943" watchObservedRunningTime="2026-09-09 23:10:50.540172896 +0000 UTC m=+711.290757822" Sep 09 23:10:51.525333 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:51.525299 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rn2fg" Sep 09 23:10:52.471712 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:52.471685 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sp7hl" Sep 09 23:10:54.482937 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:10:54.482906 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-qxxdk" Sep 09 23:11:00.523817 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:00.523782 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv" Sep 09 23:11:02.533208 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.533170 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rn2fg" Sep 09 23:11:02.649921 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.649883 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-5nfmr"] Sep 09 23:11:02.653600 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.653573 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-5nfmr" Sep 09 23:11:02.669809 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.669782 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-5nfmr"] Sep 09 23:11:02.731837 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.731803 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv"] Sep 09 23:11:02.732011 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.731994 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv" podUID="0482c1e4-668b-4546-839c-d5c99ceb65a1" containerName="manager" containerID="cri-o://4b0dc7d830adc2bfac4f6fb3c8d99ab361b0bca3f2da34455d92fdf5e2d82b8e" gracePeriod=2 Sep 09 23:11:02.744844 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.744809 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a7025a8-a3be-49dd-b57b-492487e7a8e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-5nfmr\" (UID: \"2a7025a8-a3be-49dd-b57b-492487e7a8e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-5nfmr" Sep 09 23:11:02.744984 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.744960 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cznq6\" (UniqueName: \"kubernetes.io/projected/2a7025a8-a3be-49dd-b57b-492487e7a8e3-kube-api-access-cznq6\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-5nfmr\" (UID: \"2a7025a8-a3be-49dd-b57b-492487e7a8e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-5nfmr" Sep 09 23:11:02.747130 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.747108 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv"] Sep 09 23:11:02.756876 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:02.756845 2578 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0482c1e4-668b-4546-839c-d5c99ceb65a1\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-hm6xv\": pods \"kuadrant-operator-controller-manager-76f8f686bf-hm6xv\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv" Sep 09 23:11:02.759565 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.759543 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-5nfmr"] Sep 09 23:11:02.759864 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:02.759841 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-cznq6], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-5nfmr" podUID="2a7025a8-a3be-49dd-b57b-492487e7a8e3" Sep 09 23:11:02.765558 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.765535 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n5vq5"] Sep 09 23:11:02.765951 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.765930 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0482c1e4-668b-4546-839c-d5c99ceb65a1" containerName="manager" Sep 09 23:11:02.765951 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.765946 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0482c1e4-668b-4546-839c-d5c99ceb65a1" containerName="manager" Sep 09 23:11:02.766118 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.766014 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0482c1e4-668b-4546-839c-d5c99ceb65a1" containerName="manager" Sep 09 23:11:02.769143 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.769123 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n5vq5" Sep 09 23:11:02.774513 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:02.774480 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-hm6xv\" is forbidden: User \"system:node:ip-10-0-139-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-252.ec2.internal' and this object" podUID="0482c1e4-668b-4546-839c-d5c99ceb65a1" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv" Sep 09 23:11:02.777204 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.777179 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-5nfmr"] Sep 09 23:11:02.787805 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.787736 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-5nfmr"] Sep 09 23:11:02.804576 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.804548 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n5vq5"] Sep 09 23:11:02.813833 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.813806 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rn2fg"] Sep 09 23:11:02.814051 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.814024 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rn2fg" podUID="189277a5-595d-4c84-bf48-81247b31983b" containerName="manager" containerID="cri-o://42dc23aaf80c5fa4f8ce7134636a1b851d3cc285f14dd84bc759f6570ed5ee7e" gracePeriod=2 Sep 09 23:11:02.829800 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.829772 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rn2fg"] Sep 09 23:11:02.835192 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:02.835158 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-hm6xv\" is forbidden: User \"system:node:ip-10-0-139-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-252.ec2.internal' and this object" podUID="0482c1e4-668b-4546-839c-d5c99ceb65a1" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv" Sep 09 23:11:02.837513 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:02.837486 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rn2fg\" is forbidden: User \"system:node:ip-10-0-139-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-252.ec2.internal' and this object" podUID="189277a5-595d-4c84-bf48-81247b31983b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rn2fg" Sep 09 23:11:02.846019 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.845992 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a7025a8-a3be-49dd-b57b-492487e7a8e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-5nfmr\" (UID: \"2a7025a8-a3be-49dd-b57b-492487e7a8e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-5nfmr" Sep 09 23:11:02.846155 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.846102 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cznq6\" (UniqueName: \"kubernetes.io/projected/2a7025a8-a3be-49dd-b57b-492487e7a8e3-kube-api-access-cznq6\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-5nfmr\" (UID: \"2a7025a8-a3be-49dd-b57b-492487e7a8e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-5nfmr" Sep 09 23:11:02.846490 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.846461 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a7025a8-a3be-49dd-b57b-492487e7a8e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-5nfmr\" (UID: \"2a7025a8-a3be-49dd-b57b-492487e7a8e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-5nfmr" Sep 09 23:11:02.851265 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:02.851240 2578 projected.go:196] Error preparing data for projected volume kube-api-access-cznq6 for pod kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-5nfmr: failed to fetch token: pod "kuadrant-operator-controller-manager-7978cbdfbc-5nfmr" not found Sep 09 23:11:02.851419 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:02.851306 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a7025a8-a3be-49dd-b57b-492487e7a8e3-kube-api-access-cznq6 podName:2a7025a8-a3be-49dd-b57b-492487e7a8e3 nodeName:}" failed. No retries permitted until 2026-09-09 23:11:03.351292078 +0000 UTC m=+724.101876984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cznq6" (UniqueName: "kubernetes.io/projected/2a7025a8-a3be-49dd-b57b-492487e7a8e3-kube-api-access-cznq6") pod "kuadrant-operator-controller-manager-7978cbdfbc-5nfmr" (UID: "2a7025a8-a3be-49dd-b57b-492487e7a8e3") : failed to fetch token: pod "kuadrant-operator-controller-manager-7978cbdfbc-5nfmr" not found Sep 09 23:11:02.852216 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.852199 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8k2zw"] Sep 09 23:11:02.852724 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.852708 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="189277a5-595d-4c84-bf48-81247b31983b" containerName="manager" Sep 09 23:11:02.852724 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.852725 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="189277a5-595d-4c84-bf48-81247b31983b" containerName="manager" Sep 09 23:11:02.852850 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.852799 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="189277a5-595d-4c84-bf48-81247b31983b" containerName="manager" Sep 09 23:11:02.855563 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.855547 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lhm6j"] Sep 09 23:11:02.855672 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.855653 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8k2zw" Sep 09 23:11:02.857882 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:02.857847 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-hm6xv\" is forbidden: User \"system:node:ip-10-0-139-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-252.ec2.internal' and this object" podUID="0482c1e4-668b-4546-839c-d5c99ceb65a1" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv" Sep 09 23:11:02.861096 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:02.861073 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rn2fg\" is forbidden: User \"system:node:ip-10-0-139-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-252.ec2.internal' and this object" podUID="189277a5-595d-4c84-bf48-81247b31983b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rn2fg" Sep 09 23:11:02.863537 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.863519 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lhm6j" Sep 09 23:11:02.873430 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.873400 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8k2zw"] Sep 09 23:11:02.880001 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.879969 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lhm6j"] Sep 09 23:11:02.894468 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:02.894433 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-hm6xv\" is forbidden: User \"system:node:ip-10-0-139-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-252.ec2.internal' and this object" podUID="0482c1e4-668b-4546-839c-d5c99ceb65a1" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv" Sep 09 23:11:02.946721 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.946686 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ssl5\" (UniqueName: \"kubernetes.io/projected/c109c006-0720-4ca9-a160-fe4b3bb96349-kube-api-access-6ssl5\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lhm6j\" (UID: \"c109c006-0720-4ca9-a160-fe4b3bb96349\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lhm6j" Sep 09 23:11:02.946899 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.946791 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c109c006-0720-4ca9-a160-fe4b3bb96349-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lhm6j\" (UID: \"c109c006-0720-4ca9-a160-fe4b3bb96349\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lhm6j" Sep 09 23:11:02.946899 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.946877 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7e72af8-f97a-4373-898a-63ff86e506a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-n5vq5\" (UID: \"a7e72af8-f97a-4373-898a-63ff86e506a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n5vq5" Sep 09 23:11:02.947017 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.946934 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrxs9\" (UniqueName: \"kubernetes.io/projected/a7e72af8-f97a-4373-898a-63ff86e506a2-kube-api-access-hrxs9\") pod \"kuadrant-operator-controller-manager-76f8f686bf-n5vq5\" (UID: \"a7e72af8-f97a-4373-898a-63ff86e506a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n5vq5" Sep 09 23:11:02.947017 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:02.947003 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k97mc\" (UniqueName: \"kubernetes.io/projected/b6c996fa-1326-4b5d-bca2-102b32a3b791-kube-api-access-k97mc\") pod \"limitador-operator-controller-manager-74495c69f8-8k2zw\" (UID: \"b6c996fa-1326-4b5d-bca2-102b32a3b791\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8k2zw" Sep 09 23:11:02.950447 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:02.950405 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rn2fg\" is forbidden: User \"system:node:ip-10-0-139-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-252.ec2.internal' and this object" podUID="189277a5-595d-4c84-bf48-81247b31983b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rn2fg" Sep 09 23:11:03.048273 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.048249 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k97mc\" (UniqueName: \"kubernetes.io/projected/b6c996fa-1326-4b5d-bca2-102b32a3b791-kube-api-access-k97mc\") pod \"limitador-operator-controller-manager-74495c69f8-8k2zw\" (UID: \"b6c996fa-1326-4b5d-bca2-102b32a3b791\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8k2zw" Sep 09 23:11:03.048366 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.048292 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ssl5\" (UniqueName: \"kubernetes.io/projected/c109c006-0720-4ca9-a160-fe4b3bb96349-kube-api-access-6ssl5\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lhm6j\" (UID: \"c109c006-0720-4ca9-a160-fe4b3bb96349\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lhm6j" Sep 09 23:11:03.048366 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.048324 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c109c006-0720-4ca9-a160-fe4b3bb96349-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lhm6j\" (UID: \"c109c006-0720-4ca9-a160-fe4b3bb96349\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lhm6j" Sep 09 23:11:03.048366 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.048359 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7e72af8-f97a-4373-898a-63ff86e506a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-n5vq5\" (UID: \"a7e72af8-f97a-4373-898a-63ff86e506a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n5vq5" Sep 09 23:11:03.048519 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.048439 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrxs9\" (UniqueName: \"kubernetes.io/projected/a7e72af8-f97a-4373-898a-63ff86e506a2-kube-api-access-hrxs9\") pod \"kuadrant-operator-controller-manager-76f8f686bf-n5vq5\" (UID: \"a7e72af8-f97a-4373-898a-63ff86e506a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n5vq5" Sep 09 23:11:03.048722 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.048704 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c109c006-0720-4ca9-a160-fe4b3bb96349-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lhm6j\" (UID: \"c109c006-0720-4ca9-a160-fe4b3bb96349\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lhm6j" Sep 09 23:11:03.048850 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.048830 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7e72af8-f97a-4373-898a-63ff86e506a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-n5vq5\" (UID: \"a7e72af8-f97a-4373-898a-63ff86e506a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n5vq5" Sep 09 23:11:03.057895 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.057862 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv" Sep 09 23:11:03.060457 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:03.060423 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-hm6xv\" is forbidden: User \"system:node:ip-10-0-139-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-252.ec2.internal' and this object" podUID="0482c1e4-668b-4546-839c-d5c99ceb65a1" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv" Sep 09 23:11:03.060850 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.060831 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rn2fg" Sep 09 23:11:03.062546 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:03.062452 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rn2fg\" is forbidden: User \"system:node:ip-10-0-139-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-252.ec2.internal' and this object" podUID="189277a5-595d-4c84-bf48-81247b31983b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rn2fg" Sep 09 23:11:03.064329 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.064302 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ssl5\" (UniqueName: \"kubernetes.io/projected/c109c006-0720-4ca9-a160-fe4b3bb96349-kube-api-access-6ssl5\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lhm6j\" (UID: \"c109c006-0720-4ca9-a160-fe4b3bb96349\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lhm6j" Sep 09 23:11:03.064447 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.064425 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrxs9\" (UniqueName: \"kubernetes.io/projected/a7e72af8-f97a-4373-898a-63ff86e506a2-kube-api-access-hrxs9\") pod \"kuadrant-operator-controller-manager-76f8f686bf-n5vq5\" (UID: \"a7e72af8-f97a-4373-898a-63ff86e506a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n5vq5" Sep 09 23:11:03.064554 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:03.064534 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rn2fg\" is forbidden: User \"system:node:ip-10-0-139-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-252.ec2.internal' and this object" podUID="189277a5-595d-4c84-bf48-81247b31983b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rn2fg" Sep 09 23:11:03.064657 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.064642 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k97mc\" (UniqueName: \"kubernetes.io/projected/b6c996fa-1326-4b5d-bca2-102b32a3b791-kube-api-access-k97mc\") pod \"limitador-operator-controller-manager-74495c69f8-8k2zw\" (UID: \"b6c996fa-1326-4b5d-bca2-102b32a3b791\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8k2zw" Sep 09 23:11:03.066305 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:03.066286 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-hm6xv\" is forbidden: User \"system:node:ip-10-0-139-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-252.ec2.internal' and this object" podUID="0482c1e4-668b-4546-839c-d5c99ceb65a1" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv" Sep 09 23:11:03.114217 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.114187 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n5vq5" Sep 09 23:11:03.149532 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.149497 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0482c1e4-668b-4546-839c-d5c99ceb65a1-extensions-socket-volume\") pod \"0482c1e4-668b-4546-839c-d5c99ceb65a1\" (UID: \"0482c1e4-668b-4546-839c-d5c99ceb65a1\") " Sep 09 23:11:03.149710 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.149560 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4x5sc\" (UniqueName: \"kubernetes.io/projected/0482c1e4-668b-4546-839c-d5c99ceb65a1-kube-api-access-4x5sc\") pod \"0482c1e4-668b-4546-839c-d5c99ceb65a1\" (UID: \"0482c1e4-668b-4546-839c-d5c99ceb65a1\") " Sep 09 23:11:03.149710 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.149644 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bj2jb\" (UniqueName: \"kubernetes.io/projected/189277a5-595d-4c84-bf48-81247b31983b-kube-api-access-bj2jb\") pod \"189277a5-595d-4c84-bf48-81247b31983b\" (UID: \"189277a5-595d-4c84-bf48-81247b31983b\") " Sep 09 23:11:03.149875 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.149850 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0482c1e4-668b-4546-839c-d5c99ceb65a1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0482c1e4-668b-4546-839c-d5c99ceb65a1" (UID: "0482c1e4-668b-4546-839c-d5c99ceb65a1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:11:03.150002 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.149980 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0482c1e4-668b-4546-839c-d5c99ceb65a1-extensions-socket-volume\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:11:03.152328 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.152204 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0482c1e4-668b-4546-839c-d5c99ceb65a1-kube-api-access-4x5sc" (OuterVolumeSpecName: "kube-api-access-4x5sc") pod "0482c1e4-668b-4546-839c-d5c99ceb65a1" (UID: "0482c1e4-668b-4546-839c-d5c99ceb65a1"). InnerVolumeSpecName "kube-api-access-4x5sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:11:03.152472 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.152407 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189277a5-595d-4c84-bf48-81247b31983b-kube-api-access-bj2jb" (OuterVolumeSpecName: "kube-api-access-bj2jb") pod "189277a5-595d-4c84-bf48-81247b31983b" (UID: "189277a5-595d-4c84-bf48-81247b31983b"). InnerVolumeSpecName "kube-api-access-bj2jb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:11:03.186865 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.186838 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8k2zw" Sep 09 23:11:03.194113 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.194081 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lhm6j" Sep 09 23:11:03.251503 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.251351 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x5sc\" (UniqueName: \"kubernetes.io/projected/0482c1e4-668b-4546-839c-d5c99ceb65a1-kube-api-access-4x5sc\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:11:03.251503 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.251424 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj2jb\" (UniqueName: \"kubernetes.io/projected/189277a5-595d-4c84-bf48-81247b31983b-kube-api-access-bj2jb\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:11:03.352145 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.352056 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cznq6\" (UniqueName: \"kubernetes.io/projected/2a7025a8-a3be-49dd-b57b-492487e7a8e3-kube-api-access-cznq6\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-5nfmr\" (UID: \"2a7025a8-a3be-49dd-b57b-492487e7a8e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-5nfmr" Sep 09 23:11:03.354650 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:03.354622 2578 projected.go:196] Error preparing data for projected volume kube-api-access-cznq6 for pod kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-5nfmr: failed to fetch token: pod "kuadrant-operator-controller-manager-7978cbdfbc-5nfmr" not found Sep 09 23:11:03.354812 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:03.354690 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a7025a8-a3be-49dd-b57b-492487e7a8e3-kube-api-access-cznq6 podName:2a7025a8-a3be-49dd-b57b-492487e7a8e3 nodeName:}" failed. No retries permitted until 2026-09-09 23:11:04.354673755 +0000 UTC m=+725.105258660 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cznq6" (UniqueName: "kubernetes.io/projected/2a7025a8-a3be-49dd-b57b-492487e7a8e3-kube-api-access-cznq6") pod "kuadrant-operator-controller-manager-7978cbdfbc-5nfmr" (UID: "2a7025a8-a3be-49dd-b57b-492487e7a8e3") : failed to fetch token: pod "kuadrant-operator-controller-manager-7978cbdfbc-5nfmr" not found Sep 09 23:11:03.397426 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.397371 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n5vq5"] Sep 09 23:11:03.399910 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:11:03.399864 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e72af8_f97a_4373_898a_63ff86e506a2.slice/crio-0bb755b13d535215785cab6d393afcb514246a19c943c9d6e2c4f3596135a14f WatchSource:0}: Error finding container 0bb755b13d535215785cab6d393afcb514246a19c943c9d6e2c4f3596135a14f: Status 404 returned error can't find the container with id 0bb755b13d535215785cab6d393afcb514246a19c943c9d6e2c4f3596135a14f Sep 09 23:11:03.470992 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.470969 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8k2zw"] Sep 09 23:11:03.474091 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:11:03.474064 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c996fa_1326_4b5d_bca2_102b32a3b791.slice/crio-3801bed8f6cfa18f73361db72166441492f9699f5dfa7fa3eb6fa7a19b5487ef WatchSource:0}: Error finding container 3801bed8f6cfa18f73361db72166441492f9699f5dfa7fa3eb6fa7a19b5487ef: Status 404 returned error can't find the container with id 3801bed8f6cfa18f73361db72166441492f9699f5dfa7fa3eb6fa7a19b5487ef Sep 09 23:11:03.474817 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.474796 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lhm6j"] Sep 09 23:11:03.476611 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:11:03.476591 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc109c006_0720_4ca9_a160_fe4b3bb96349.slice/crio-85d5e51e9e22d1692ed509070bae6ec39c9b82ebc19cf39be37810e35bab2058 WatchSource:0}: Error finding container 85d5e51e9e22d1692ed509070bae6ec39c9b82ebc19cf39be37810e35bab2058: Status 404 returned error can't find the container with id 85d5e51e9e22d1692ed509070bae6ec39c9b82ebc19cf39be37810e35bab2058 Sep 09 23:11:03.585046 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.585005 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lhm6j" event={"ID":"c109c006-0720-4ca9-a160-fe4b3bb96349","Type":"ContainerStarted","Data":"876c720ace981145029207890bafa7a1ef6f380e55ddd23eeb00f8f63b8676fc"} Sep 09 23:11:03.585511 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.585080 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lhm6j" Sep 09 23:11:03.585511 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.585107 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lhm6j" event={"ID":"c109c006-0720-4ca9-a160-fe4b3bb96349","Type":"ContainerStarted","Data":"85d5e51e9e22d1692ed509070bae6ec39c9b82ebc19cf39be37810e35bab2058"} Sep 09 23:11:03.586455 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.586432 2578 generic.go:356] "Generic (PLEG): container finished" podID="189277a5-595d-4c84-bf48-81247b31983b" containerID="42dc23aaf80c5fa4f8ce7134636a1b851d3cc285f14dd84bc759f6570ed5ee7e" exitCode=0 Sep 09 23:11:03.586566 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.586469 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rn2fg" Sep 09 23:11:03.586566 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.586507 2578 scope.go:117] "RemoveContainer" containerID="42dc23aaf80c5fa4f8ce7134636a1b851d3cc285f14dd84bc759f6570ed5ee7e" Sep 09 23:11:03.587326 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:03.587296 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-hm6xv\" is forbidden: User \"system:node:ip-10-0-139-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-252.ec2.internal' and this object" podUID="0482c1e4-668b-4546-839c-d5c99ceb65a1" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv" Sep 09 23:11:03.588310 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.588277 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n5vq5" event={"ID":"a7e72af8-f97a-4373-898a-63ff86e506a2","Type":"ContainerStarted","Data":"108a4a5ccf87fb69d34d29700b8d6bebe9b944cb04660acea1acbbdac92b8e0c"} Sep 09 23:11:03.588491 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.588330 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n5vq5" Sep 09 23:11:03.588491 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.588355 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n5vq5" event={"ID":"a7e72af8-f97a-4373-898a-63ff86e506a2","Type":"ContainerStarted","Data":"0bb755b13d535215785cab6d393afcb514246a19c943c9d6e2c4f3596135a14f"} Sep 09 23:11:03.589885 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.589862 2578 generic.go:356] "Generic (PLEG): container finished" podID="0482c1e4-668b-4546-839c-d5c99ceb65a1" containerID="4b0dc7d830adc2bfac4f6fb3c8d99ab361b0bca3f2da34455d92fdf5e2d82b8e" exitCode=0 Sep 09 23:11:03.589959 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.589903 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv" Sep 09 23:11:03.592049 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.591912 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-5nfmr" Sep 09 23:11:03.592049 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.591925 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8k2zw" event={"ID":"b6c996fa-1326-4b5d-bca2-102b32a3b791","Type":"ContainerStarted","Data":"45e7ecd3619df536c47a8cb5c6799bc4d1bd51feaab07696c042c9e92774e56b"} Sep 09 23:11:03.592049 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.591966 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8k2zw" event={"ID":"b6c996fa-1326-4b5d-bca2-102b32a3b791","Type":"ContainerStarted","Data":"3801bed8f6cfa18f73361db72166441492f9699f5dfa7fa3eb6fa7a19b5487ef"} Sep 09 23:11:03.592455 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.592071 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8k2zw" Sep 09 23:11:03.596184 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.596165 2578 scope.go:117] "RemoveContainer" containerID="42dc23aaf80c5fa4f8ce7134636a1b851d3cc285f14dd84bc759f6570ed5ee7e" Sep 09 23:11:03.596523 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:03.596449 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42dc23aaf80c5fa4f8ce7134636a1b851d3cc285f14dd84bc759f6570ed5ee7e\": container with ID starting with 42dc23aaf80c5fa4f8ce7134636a1b851d3cc285f14dd84bc759f6570ed5ee7e not found: ID does not exist" containerID="42dc23aaf80c5fa4f8ce7134636a1b851d3cc285f14dd84bc759f6570ed5ee7e" Sep 09 23:11:03.596523 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.596486 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42dc23aaf80c5fa4f8ce7134636a1b851d3cc285f14dd84bc759f6570ed5ee7e"} err="failed to get container status \"42dc23aaf80c5fa4f8ce7134636a1b851d3cc285f14dd84bc759f6570ed5ee7e\": rpc error: code = NotFound desc = could not find container \"42dc23aaf80c5fa4f8ce7134636a1b851d3cc285f14dd84bc759f6570ed5ee7e\": container with ID starting with 42dc23aaf80c5fa4f8ce7134636a1b851d3cc285f14dd84bc759f6570ed5ee7e not found: ID does not exist" Sep 09 23:11:03.596523 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.596510 2578 scope.go:117] "RemoveContainer" containerID="4b0dc7d830adc2bfac4f6fb3c8d99ab361b0bca3f2da34455d92fdf5e2d82b8e" Sep 09 23:11:03.600736 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.600720 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-5nfmr" Sep 09 23:11:03.604796 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.604782 2578 scope.go:117] "RemoveContainer" containerID="4b0dc7d830adc2bfac4f6fb3c8d99ab361b0bca3f2da34455d92fdf5e2d82b8e" Sep 09 23:11:03.605058 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:03.605043 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b0dc7d830adc2bfac4f6fb3c8d99ab361b0bca3f2da34455d92fdf5e2d82b8e\": container with ID starting with 4b0dc7d830adc2bfac4f6fb3c8d99ab361b0bca3f2da34455d92fdf5e2d82b8e not found: ID does not exist" containerID="4b0dc7d830adc2bfac4f6fb3c8d99ab361b0bca3f2da34455d92fdf5e2d82b8e" Sep 09 23:11:03.605109 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.605065 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b0dc7d830adc2bfac4f6fb3c8d99ab361b0bca3f2da34455d92fdf5e2d82b8e"} err="failed to get container status \"4b0dc7d830adc2bfac4f6fb3c8d99ab361b0bca3f2da34455d92fdf5e2d82b8e\": rpc error: code = NotFound desc = could not find container \"4b0dc7d830adc2bfac4f6fb3c8d99ab361b0bca3f2da34455d92fdf5e2d82b8e\": container with ID starting with 4b0dc7d830adc2bfac4f6fb3c8d99ab361b0bca3f2da34455d92fdf5e2d82b8e not found: ID does not exist" Sep 09 23:11:03.607476 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.607422 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lhm6j" podStartSLOduration=1.607409036 podStartE2EDuration="1.607409036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:11:03.60631581 +0000 UTC m=+724.356900737" watchObservedRunningTime="2026-09-09 23:11:03.607409036 +0000 UTC m=+724.357993960" Sep 09 23:11:03.608311 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:03.608284 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rn2fg\" is forbidden: User \"system:node:ip-10-0-139-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-252.ec2.internal' and this object" podUID="189277a5-595d-4c84-bf48-81247b31983b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rn2fg" Sep 09 23:11:03.634601 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.634564 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n5vq5" podStartSLOduration=1.634550753 podStartE2EDuration="1.634550753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:11:03.632914074 +0000 UTC m=+724.383499000" watchObservedRunningTime="2026-09-09 23:11:03.634550753 +0000 UTC m=+724.385135680" Sep 09 23:11:03.635011 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:03.634972 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-hm6xv\" is forbidden: User \"system:node:ip-10-0-139-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-252.ec2.internal' and this object" podUID="0482c1e4-668b-4546-839c-d5c99ceb65a1" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hm6xv" Sep 09 23:11:03.637585 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:03.637562 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-5nfmr\" is forbidden: User \"system:node:ip-10-0-139-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-252.ec2.internal' and this object" podUID="2a7025a8-a3be-49dd-b57b-492487e7a8e3" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-5nfmr" Sep 09 23:11:03.639460 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:03.639404 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rn2fg\" is forbidden: User \"system:node:ip-10-0-139-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-252.ec2.internal' and this object" podUID="189277a5-595d-4c84-bf48-81247b31983b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rn2fg" Sep 09 23:11:03.658819 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.658769 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8k2zw" podStartSLOduration=1.658757445 podStartE2EDuration="1.658757445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:11:03.657233033 +0000 UTC m=+724.407817960" watchObservedRunningTime="2026-09-09 23:11:03.658757445 +0000 UTC m=+724.409342372" Sep 09 23:11:03.755216 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.755178 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a7025a8-a3be-49dd-b57b-492487e7a8e3-extensions-socket-volume\") pod \"2a7025a8-a3be-49dd-b57b-492487e7a8e3\" (UID: \"2a7025a8-a3be-49dd-b57b-492487e7a8e3\") " Sep 09 23:11:03.755427 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.755404 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cznq6\" (UniqueName: \"kubernetes.io/projected/2a7025a8-a3be-49dd-b57b-492487e7a8e3-kube-api-access-cznq6\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:11:03.755539 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.755512 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7025a8-a3be-49dd-b57b-492487e7a8e3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2a7025a8-a3be-49dd-b57b-492487e7a8e3" (UID: "2a7025a8-a3be-49dd-b57b-492487e7a8e3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:11:03.852471 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.852439 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0482c1e4-668b-4546-839c-d5c99ceb65a1" path="/var/lib/kubelet/pods/0482c1e4-668b-4546-839c-d5c99ceb65a1/volumes" Sep 09 23:11:03.852763 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.852751 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="189277a5-595d-4c84-bf48-81247b31983b" path="/var/lib/kubelet/pods/189277a5-595d-4c84-bf48-81247b31983b/volumes" Sep 09 23:11:03.853021 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.853010 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a7025a8-a3be-49dd-b57b-492487e7a8e3" path="/var/lib/kubelet/pods/2a7025a8-a3be-49dd-b57b-492487e7a8e3/volumes" Sep 09 23:11:03.855767 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:03.855723 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a7025a8-a3be-49dd-b57b-492487e7a8e3-extensions-socket-volume\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:11:04.605433 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:04.605402 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-5nfmr" Sep 09 23:11:04.609562 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:04.609535 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-5nfmr\" is forbidden: User \"system:node:ip-10-0-139-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-252.ec2.internal' and this object" podUID="2a7025a8-a3be-49dd-b57b-492487e7a8e3" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-5nfmr" Sep 09 23:11:14.608007 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:14.607975 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n5vq5" Sep 09 23:11:14.608550 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:14.608053 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8k2zw" Sep 09 23:11:14.608550 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:14.608203 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lhm6j" Sep 09 23:11:14.704002 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:14.703970 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n5vq5"] Sep 09 23:11:14.704212 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:14.704187 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n5vq5" podUID="a7e72af8-f97a-4373-898a-63ff86e506a2" containerName="manager" containerID="cri-o://108a4a5ccf87fb69d34d29700b8d6bebe9b944cb04660acea1acbbdac92b8e0c" gracePeriod=10 Sep 09 23:11:14.992088 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:14.992064 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n5vq5" Sep 09 23:11:15.052357 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:15.052321 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrxs9\" (UniqueName: \"kubernetes.io/projected/a7e72af8-f97a-4373-898a-63ff86e506a2-kube-api-access-hrxs9\") pod \"a7e72af8-f97a-4373-898a-63ff86e506a2\" (UID: \"a7e72af8-f97a-4373-898a-63ff86e506a2\") " Sep 09 23:11:15.052544 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:15.052372 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7e72af8-f97a-4373-898a-63ff86e506a2-extensions-socket-volume\") pod \"a7e72af8-f97a-4373-898a-63ff86e506a2\" (UID: \"a7e72af8-f97a-4373-898a-63ff86e506a2\") " Sep 09 23:11:15.052813 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:15.052790 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e72af8-f97a-4373-898a-63ff86e506a2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a7e72af8-f97a-4373-898a-63ff86e506a2" (UID: "a7e72af8-f97a-4373-898a-63ff86e506a2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:11:15.054528 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:15.054500 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e72af8-f97a-4373-898a-63ff86e506a2-kube-api-access-hrxs9" (OuterVolumeSpecName: "kube-api-access-hrxs9") pod "a7e72af8-f97a-4373-898a-63ff86e506a2" (UID: "a7e72af8-f97a-4373-898a-63ff86e506a2"). InnerVolumeSpecName "kube-api-access-hrxs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:11:15.153265 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:15.153189 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrxs9\" (UniqueName: \"kubernetes.io/projected/a7e72af8-f97a-4373-898a-63ff86e506a2-kube-api-access-hrxs9\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:11:15.153265 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:15.153216 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7e72af8-f97a-4373-898a-63ff86e506a2-extensions-socket-volume\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:11:15.662043 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:15.662005 2578 generic.go:356] "Generic (PLEG): container finished" podID="a7e72af8-f97a-4373-898a-63ff86e506a2" containerID="108a4a5ccf87fb69d34d29700b8d6bebe9b944cb04660acea1acbbdac92b8e0c" exitCode=0 Sep 09 23:11:15.662466 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:15.662056 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n5vq5" Sep 09 23:11:15.662466 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:15.662091 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n5vq5" event={"ID":"a7e72af8-f97a-4373-898a-63ff86e506a2","Type":"ContainerDied","Data":"108a4a5ccf87fb69d34d29700b8d6bebe9b944cb04660acea1acbbdac92b8e0c"} Sep 09 23:11:15.662466 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:15.662165 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n5vq5" event={"ID":"a7e72af8-f97a-4373-898a-63ff86e506a2","Type":"ContainerDied","Data":"0bb755b13d535215785cab6d393afcb514246a19c943c9d6e2c4f3596135a14f"} Sep 09 23:11:15.662466 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:15.662189 2578 scope.go:117] "RemoveContainer" containerID="108a4a5ccf87fb69d34d29700b8d6bebe9b944cb04660acea1acbbdac92b8e0c" Sep 09 23:11:15.685697 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:15.673040 2578 scope.go:117] "RemoveContainer" containerID="108a4a5ccf87fb69d34d29700b8d6bebe9b944cb04660acea1acbbdac92b8e0c" Sep 09 23:11:15.685697 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:15.685594 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"108a4a5ccf87fb69d34d29700b8d6bebe9b944cb04660acea1acbbdac92b8e0c\": container with ID starting with 108a4a5ccf87fb69d34d29700b8d6bebe9b944cb04660acea1acbbdac92b8e0c not found: ID does not exist" containerID="108a4a5ccf87fb69d34d29700b8d6bebe9b944cb04660acea1acbbdac92b8e0c" Sep 09 23:11:15.685697 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:15.685657 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"108a4a5ccf87fb69d34d29700b8d6bebe9b944cb04660acea1acbbdac92b8e0c"} err="failed to get container status \"108a4a5ccf87fb69d34d29700b8d6bebe9b944cb04660acea1acbbdac92b8e0c\": rpc error: code = NotFound desc = could not find container \"108a4a5ccf87fb69d34d29700b8d6bebe9b944cb04660acea1acbbdac92b8e0c\": container with ID starting with 108a4a5ccf87fb69d34d29700b8d6bebe9b944cb04660acea1acbbdac92b8e0c not found: ID does not exist" Sep 09 23:11:15.697738 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:15.697708 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n5vq5"] Sep 09 23:11:15.713350 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:15.713320 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n5vq5"] Sep 09 23:11:15.852074 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:15.852044 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7e72af8-f97a-4373-898a-63ff86e506a2" path="/var/lib/kubelet/pods/a7e72af8-f97a-4373-898a-63ff86e506a2/volumes" Sep 09 23:11:31.009287 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.009236 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s"] Sep 09 23:11:31.009788 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.009659 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7e72af8-f97a-4373-898a-63ff86e506a2" containerName="manager" Sep 09 23:11:31.009788 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.009672 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7e72af8-f97a-4373-898a-63ff86e506a2" containerName="manager" Sep 09 23:11:31.009788 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.009746 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7e72af8-f97a-4373-898a-63ff86e506a2" containerName="manager" Sep 09 23:11:31.011834 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.011815 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.014421 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.014396 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9svwb\"" Sep 09 23:11:31.014558 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.014483 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 09 23:11:31.027455 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.027434 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s"] Sep 09 23:11:31.090342 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.090311 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a051327-0bb5-4299-a04e-b402de10abf8-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.090537 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.090340 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a051327-0bb5-4299-a04e-b402de10abf8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.090537 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.090480 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a051327-0bb5-4299-a04e-b402de10abf8-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.090537 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.090522 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a051327-0bb5-4299-a04e-b402de10abf8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.090706 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.090548 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a051327-0bb5-4299-a04e-b402de10abf8-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.090706 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.090588 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a051327-0bb5-4299-a04e-b402de10abf8-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.090706 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.090637 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a051327-0bb5-4299-a04e-b402de10abf8-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.090706 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.090657 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvdtv\" (UniqueName: \"kubernetes.io/projected/1a051327-0bb5-4299-a04e-b402de10abf8-kube-api-access-tvdtv\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.090906 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.090713 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a051327-0bb5-4299-a04e-b402de10abf8-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.191208 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.191174 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a051327-0bb5-4299-a04e-b402de10abf8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.191418 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.191214 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a051327-0bb5-4299-a04e-b402de10abf8-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.191418 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.191238 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a051327-0bb5-4299-a04e-b402de10abf8-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.191418 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.191263 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a051327-0bb5-4299-a04e-b402de10abf8-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.191418 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.191400 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvdtv\" (UniqueName: \"kubernetes.io/projected/1a051327-0bb5-4299-a04e-b402de10abf8-kube-api-access-tvdtv\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.191637 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.191458 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a051327-0bb5-4299-a04e-b402de10abf8-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.191637 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.191533 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a051327-0bb5-4299-a04e-b402de10abf8-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.191637 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.191558 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a051327-0bb5-4299-a04e-b402de10abf8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.191637 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.191631 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a051327-0bb5-4299-a04e-b402de10abf8-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.191841 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.191811 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a051327-0bb5-4299-a04e-b402de10abf8-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.192437 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.192112 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a051327-0bb5-4299-a04e-b402de10abf8-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.192437 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.192113 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a051327-0bb5-4299-a04e-b402de10abf8-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.192437 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.192365 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a051327-0bb5-4299-a04e-b402de10abf8-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.192683 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.192527 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a051327-0bb5-4299-a04e-b402de10abf8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.194104 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.194085 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a051327-0bb5-4299-a04e-b402de10abf8-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.194488 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.194472 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a051327-0bb5-4299-a04e-b402de10abf8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.199592 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.199567 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a051327-0bb5-4299-a04e-b402de10abf8-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.199754 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.199739 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvdtv\" (UniqueName: \"kubernetes.io/projected/1a051327-0bb5-4299-a04e-b402de10abf8-kube-api-access-tvdtv\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lzj2s\" (UID: \"1a051327-0bb5-4299-a04e-b402de10abf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.322861 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.322833 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:31.598277 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.598251 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s"] Sep 09 23:11:31.600346 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:11:31.600318 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a051327_0bb5_4299_a04e_b402de10abf8.slice/crio-4d1f8cfab5cdc159f91b8c76fe01f5fbbaf80e5338edae878212085149e042bc WatchSource:0}: Error finding container 4d1f8cfab5cdc159f91b8c76fe01f5fbbaf80e5338edae878212085149e042bc: Status 404 returned error can't find the container with id 4d1f8cfab5cdc159f91b8c76fe01f5fbbaf80e5338edae878212085149e042bc Sep 09 23:11:31.744139 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:31.744102 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" event={"ID":"1a051327-0bb5-4299-a04e-b402de10abf8","Type":"ContainerStarted","Data":"4d1f8cfab5cdc159f91b8c76fe01f5fbbaf80e5338edae878212085149e042bc"} Sep 09 23:11:34.905171 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:34.905126 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 09 23:11:34.905540 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:34.905213 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 09 23:11:34.905540 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:34.905244 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 09 23:11:35.767442 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:35.767406 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" event={"ID":"1a051327-0bb5-4299-a04e-b402de10abf8","Type":"ContainerStarted","Data":"16d830779fe27fa7cba737eb91a738268c7bc02e65b8ec0b925d3ade3a5c4507"} Sep 09 23:11:35.787142 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:35.787096 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" podStartSLOduration=2.484840962 podStartE2EDuration="5.787079646s" totalImagesPullingTime="3.302238684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:11:35.785614772 +0000 UTC m=+756.536199711" watchObservedRunningTime="2026-09-09 23:11:35.787079646 +0000 UTC m=+756.537664572" Sep 09 23:11:36.323982 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:36.323950 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:36.328401 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:36.328354 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:36.772800 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:36.772768 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:36.773690 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:36.773671 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:11:36.805079 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:36.805055 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-9hrnw"] Sep 09 23:11:36.807592 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:36.807578 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-9hrnw" Sep 09 23:11:36.810735 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:36.810715 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kr69j\"" Sep 09 23:11:36.811404 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:36.811374 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 09 23:11:36.826848 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:36.826825 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-9hrnw"] Sep 09 23:11:36.925948 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:36.925919 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-9hrnw"] Sep 09 23:11:36.947513 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:36.947464 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsdv8\" (UniqueName: \"kubernetes.io/projected/fa8f1470-95b8-4638-9ce7-d439fa95c2cb-kube-api-access-tsdv8\") pod \"limitador-limitador-8588749977-9hrnw\" (UID: \"fa8f1470-95b8-4638-9ce7-d439fa95c2cb\") " pod="kuadrant-system/limitador-limitador-8588749977-9hrnw" Sep 09 23:11:36.947718 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:36.947545 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fa8f1470-95b8-4638-9ce7-d439fa95c2cb-config-file\") pod \"limitador-limitador-8588749977-9hrnw\" (UID: \"fa8f1470-95b8-4638-9ce7-d439fa95c2cb\") " pod="kuadrant-system/limitador-limitador-8588749977-9hrnw" Sep 09 23:11:37.048615 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:37.048526 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsdv8\" (UniqueName: \"kubernetes.io/projected/fa8f1470-95b8-4638-9ce7-d439fa95c2cb-kube-api-access-tsdv8\") pod \"limitador-limitador-8588749977-9hrnw\" (UID: \"fa8f1470-95b8-4638-9ce7-d439fa95c2cb\") " pod="kuadrant-system/limitador-limitador-8588749977-9hrnw" Sep 09 23:11:37.048615 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:37.048601 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fa8f1470-95b8-4638-9ce7-d439fa95c2cb-config-file\") pod \"limitador-limitador-8588749977-9hrnw\" (UID: \"fa8f1470-95b8-4638-9ce7-d439fa95c2cb\") " pod="kuadrant-system/limitador-limitador-8588749977-9hrnw" Sep 09 23:11:37.049170 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:37.049150 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fa8f1470-95b8-4638-9ce7-d439fa95c2cb-config-file\") pod \"limitador-limitador-8588749977-9hrnw\" (UID: \"fa8f1470-95b8-4638-9ce7-d439fa95c2cb\") " pod="kuadrant-system/limitador-limitador-8588749977-9hrnw" Sep 09 23:11:37.056701 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:37.056681 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsdv8\" (UniqueName: \"kubernetes.io/projected/fa8f1470-95b8-4638-9ce7-d439fa95c2cb-kube-api-access-tsdv8\") pod \"limitador-limitador-8588749977-9hrnw\" (UID: \"fa8f1470-95b8-4638-9ce7-d439fa95c2cb\") " pod="kuadrant-system/limitador-limitador-8588749977-9hrnw" Sep 09 23:11:37.117331 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:37.117303 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-9hrnw" Sep 09 23:11:37.391854 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:37.391824 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-9hrnw"] Sep 09 23:11:37.393919 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:11:37.393893 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa8f1470_95b8_4638_9ce7_d439fa95c2cb.slice/crio-851bb3dd9f87d564399dda2c581f3ae497212a924949389a1b1984ff3ac26004 WatchSource:0}: Error finding container 851bb3dd9f87d564399dda2c581f3ae497212a924949389a1b1984ff3ac26004: Status 404 returned error can't find the container with id 851bb3dd9f87d564399dda2c581f3ae497212a924949389a1b1984ff3ac26004 Sep 09 23:11:37.777199 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:37.777166 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-wsqqd"] Sep 09 23:11:37.780771 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:37.780744 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-9hrnw" event={"ID":"fa8f1470-95b8-4638-9ce7-d439fa95c2cb","Type":"ContainerStarted","Data":"851bb3dd9f87d564399dda2c581f3ae497212a924949389a1b1984ff3ac26004"} Sep 09 23:11:37.780911 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:37.780824 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-wsqqd" Sep 09 23:11:37.783448 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:37.783429 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ll5q8\"" Sep 09 23:11:37.791556 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:37.789552 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wsqqd"] Sep 09 23:11:37.855407 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:37.855362 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnsxg\" (UniqueName: \"kubernetes.io/projected/c032dd92-a7c4-4416-a4c5-a0e839650dcd-kube-api-access-gnsxg\") pod \"authorino-59c974bd48-wsqqd\" (UID: \"c032dd92-a7c4-4416-a4c5-a0e839650dcd\") " pod="kuadrant-system/authorino-59c974bd48-wsqqd" Sep 09 23:11:37.956827 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:37.956789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnsxg\" (UniqueName: \"kubernetes.io/projected/c032dd92-a7c4-4416-a4c5-a0e839650dcd-kube-api-access-gnsxg\") pod \"authorino-59c974bd48-wsqqd\" (UID: \"c032dd92-a7c4-4416-a4c5-a0e839650dcd\") " pod="kuadrant-system/authorino-59c974bd48-wsqqd" Sep 09 23:11:37.965511 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:37.965490 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnsxg\" (UniqueName: \"kubernetes.io/projected/c032dd92-a7c4-4416-a4c5-a0e839650dcd-kube-api-access-gnsxg\") pod \"authorino-59c974bd48-wsqqd\" (UID: \"c032dd92-a7c4-4416-a4c5-a0e839650dcd\") " pod="kuadrant-system/authorino-59c974bd48-wsqqd" Sep 09 23:11:38.092651 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:38.092580 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-wsqqd" Sep 09 23:11:38.362405 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:38.362359 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wsqqd"] Sep 09 23:11:38.504473 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:11:38.504444 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc032dd92_a7c4_4416_a4c5_a0e839650dcd.slice/crio-64cc86e207a85e31c7fa4b66ccdabf7d8786ba10b486dd61c2e878ad8f765aee WatchSource:0}: Error finding container 64cc86e207a85e31c7fa4b66ccdabf7d8786ba10b486dd61c2e878ad8f765aee: Status 404 returned error can't find the container with id 64cc86e207a85e31c7fa4b66ccdabf7d8786ba10b486dd61c2e878ad8f765aee Sep 09 23:11:38.790483 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:38.790442 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-9hrnw" event={"ID":"fa8f1470-95b8-4638-9ce7-d439fa95c2cb","Type":"ContainerStarted","Data":"a770ad335b63a8b8107fd3100fa72d28885c11099e8ec11d8fd695f9f6424d1a"} Sep 09 23:11:38.790681 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:38.790663 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-9hrnw" Sep 09 23:11:38.791724 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:38.791699 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-wsqqd" event={"ID":"c032dd92-a7c4-4416-a4c5-a0e839650dcd","Type":"ContainerStarted","Data":"64cc86e207a85e31c7fa4b66ccdabf7d8786ba10b486dd61c2e878ad8f765aee"} Sep 09 23:11:38.806616 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:38.806568 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-9hrnw" podStartSLOduration=1.6597325939999998 podStartE2EDuration="2.806554127s" totalImagesPullingTime="1.146821533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:11:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:11:38.804978693 +0000 UTC m=+759.555563643" watchObservedRunningTime="2026-09-09 23:11:38.806554127 +0000 UTC m=+759.557139055" Sep 09 23:11:40.804189 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:40.804155 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-wsqqd" event={"ID":"c032dd92-a7c4-4416-a4c5-a0e839650dcd","Type":"ContainerStarted","Data":"b3ca9aa0545aa60ddd7bc4deb7ee8d88fc96911c2279fea1da5bb50f7da2a65e"} Sep 09 23:11:40.818596 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:40.818551 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-wsqqd" podStartSLOduration=2.147062853 podStartE2EDuration="3.818535197s" totalImagesPullingTime="1.671472344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:11:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:11:40.817618918 +0000 UTC m=+761.568203846" watchObservedRunningTime="2026-09-09 23:11:40.818535197 +0000 UTC m=+761.569120469" Sep 09 23:11:49.798404 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:49.798349 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-9hrnw" Sep 09 23:11:53.280187 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:53.280144 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-9hrnw"] Sep 09 23:11:53.280678 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:53.280351 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-9hrnw" podUID="fa8f1470-95b8-4638-9ce7-d439fa95c2cb" containerName="limitador" containerID="cri-o://a770ad335b63a8b8107fd3100fa72d28885c11099e8ec11d8fd695f9f6424d1a" gracePeriod=30 Sep 09 23:11:53.837606 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:53.837579 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-9hrnw" Sep 09 23:11:53.881906 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:53.879356 2578 generic.go:356] "Generic (PLEG): container finished" podID="fa8f1470-95b8-4638-9ce7-d439fa95c2cb" containerID="a770ad335b63a8b8107fd3100fa72d28885c11099e8ec11d8fd695f9f6424d1a" exitCode=0 Sep 09 23:11:53.881906 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:53.879462 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-9hrnw" event={"ID":"fa8f1470-95b8-4638-9ce7-d439fa95c2cb","Type":"ContainerDied","Data":"a770ad335b63a8b8107fd3100fa72d28885c11099e8ec11d8fd695f9f6424d1a"} Sep 09 23:11:53.881906 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:53.879509 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-9hrnw" event={"ID":"fa8f1470-95b8-4638-9ce7-d439fa95c2cb","Type":"ContainerDied","Data":"851bb3dd9f87d564399dda2c581f3ae497212a924949389a1b1984ff3ac26004"} Sep 09 23:11:53.881906 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:53.879538 2578 scope.go:117] "RemoveContainer" containerID="a770ad335b63a8b8107fd3100fa72d28885c11099e8ec11d8fd695f9f6424d1a" Sep 09 23:11:53.881906 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:53.879701 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-9hrnw" Sep 09 23:11:53.890323 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:53.890302 2578 scope.go:117] "RemoveContainer" containerID="a770ad335b63a8b8107fd3100fa72d28885c11099e8ec11d8fd695f9f6424d1a" Sep 09 23:11:53.890637 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:11:53.890611 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a770ad335b63a8b8107fd3100fa72d28885c11099e8ec11d8fd695f9f6424d1a\": container with ID starting with a770ad335b63a8b8107fd3100fa72d28885c11099e8ec11d8fd695f9f6424d1a not found: ID does not exist" containerID="a770ad335b63a8b8107fd3100fa72d28885c11099e8ec11d8fd695f9f6424d1a" Sep 09 23:11:53.890692 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:53.890644 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a770ad335b63a8b8107fd3100fa72d28885c11099e8ec11d8fd695f9f6424d1a"} err="failed to get container status \"a770ad335b63a8b8107fd3100fa72d28885c11099e8ec11d8fd695f9f6424d1a\": rpc error: code = NotFound desc = could not find container \"a770ad335b63a8b8107fd3100fa72d28885c11099e8ec11d8fd695f9f6424d1a\": container with ID starting with a770ad335b63a8b8107fd3100fa72d28885c11099e8ec11d8fd695f9f6424d1a not found: ID does not exist" Sep 09 23:11:54.005891 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:54.005850 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsdv8\" (UniqueName: \"kubernetes.io/projected/fa8f1470-95b8-4638-9ce7-d439fa95c2cb-kube-api-access-tsdv8\") pod \"fa8f1470-95b8-4638-9ce7-d439fa95c2cb\" (UID: \"fa8f1470-95b8-4638-9ce7-d439fa95c2cb\") " Sep 09 23:11:54.006087 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:54.005939 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fa8f1470-95b8-4638-9ce7-d439fa95c2cb-config-file\") pod \"fa8f1470-95b8-4638-9ce7-d439fa95c2cb\" (UID: \"fa8f1470-95b8-4638-9ce7-d439fa95c2cb\") " Sep 09 23:11:54.006283 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:54.006258 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa8f1470-95b8-4638-9ce7-d439fa95c2cb-config-file" (OuterVolumeSpecName: "config-file") pod "fa8f1470-95b8-4638-9ce7-d439fa95c2cb" (UID: "fa8f1470-95b8-4638-9ce7-d439fa95c2cb"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:11:54.008095 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:54.008071 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa8f1470-95b8-4638-9ce7-d439fa95c2cb-kube-api-access-tsdv8" (OuterVolumeSpecName: "kube-api-access-tsdv8") pod "fa8f1470-95b8-4638-9ce7-d439fa95c2cb" (UID: "fa8f1470-95b8-4638-9ce7-d439fa95c2cb"). InnerVolumeSpecName "kube-api-access-tsdv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:11:54.107555 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:54.107517 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsdv8\" (UniqueName: \"kubernetes.io/projected/fa8f1470-95b8-4638-9ce7-d439fa95c2cb-kube-api-access-tsdv8\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:11:54.107555 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:54.107550 2578 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fa8f1470-95b8-4638-9ce7-d439fa95c2cb-config-file\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:11:54.200445 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:54.200406 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-9hrnw"] Sep 09 23:11:54.204348 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:54.204319 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-9hrnw"] Sep 09 23:11:55.852350 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:55.852316 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa8f1470-95b8-4638-9ce7-d439fa95c2cb" path="/var/lib/kubelet/pods/fa8f1470-95b8-4638-9ce7-d439fa95c2cb/volumes" Sep 09 23:11:58.424157 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:58.424120 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-89bdz"] Sep 09 23:11:58.424709 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:58.424689 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa8f1470-95b8-4638-9ce7-d439fa95c2cb" containerName="limitador" Sep 09 23:11:58.424753 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:58.424715 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa8f1470-95b8-4638-9ce7-d439fa95c2cb" containerName="limitador" Sep 09 23:11:58.424791 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:58.424785 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa8f1470-95b8-4638-9ce7-d439fa95c2cb" containerName="limitador" Sep 09 23:11:58.429288 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:58.429265 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-89bdz" Sep 09 23:11:58.432863 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:58.432546 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 23:11:58.432863 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:58.432611 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 09 23:11:58.432863 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:58.432547 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 09 23:11:58.432863 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:58.432828 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-jnn5d\"" Sep 09 23:11:58.434515 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:58.434483 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-89bdz"] Sep 09 23:11:58.444525 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:58.444502 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0201c758-08a3-4b63-ae0b-820e3a0577ca-data\") pod \"postgres-69fb4594cf-89bdz\" (UID: \"0201c758-08a3-4b63-ae0b-820e3a0577ca\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-89bdz" Sep 09 23:11:58.444645 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:58.444557 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsctb\" (UniqueName: \"kubernetes.io/projected/0201c758-08a3-4b63-ae0b-820e3a0577ca-kube-api-access-qsctb\") pod \"postgres-69fb4594cf-89bdz\" (UID: \"0201c758-08a3-4b63-ae0b-820e3a0577ca\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-89bdz" Sep 09 23:11:58.544988 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:58.544949 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0201c758-08a3-4b63-ae0b-820e3a0577ca-data\") pod \"postgres-69fb4594cf-89bdz\" (UID: \"0201c758-08a3-4b63-ae0b-820e3a0577ca\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-89bdz" Sep 09 23:11:58.545176 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:58.545010 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsctb\" (UniqueName: \"kubernetes.io/projected/0201c758-08a3-4b63-ae0b-820e3a0577ca-kube-api-access-qsctb\") pod \"postgres-69fb4594cf-89bdz\" (UID: \"0201c758-08a3-4b63-ae0b-820e3a0577ca\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-89bdz" Sep 09 23:11:58.545313 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:58.545292 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0201c758-08a3-4b63-ae0b-820e3a0577ca-data\") pod \"postgres-69fb4594cf-89bdz\" (UID: \"0201c758-08a3-4b63-ae0b-820e3a0577ca\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-89bdz" Sep 09 23:11:58.552833 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:58.552811 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsctb\" (UniqueName: \"kubernetes.io/projected/0201c758-08a3-4b63-ae0b-820e3a0577ca-kube-api-access-qsctb\") pod \"postgres-69fb4594cf-89bdz\" (UID: \"0201c758-08a3-4b63-ae0b-820e3a0577ca\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-89bdz" Sep 09 23:11:58.744509 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:58.744419 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-89bdz" Sep 09 23:11:59.013751 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:59.013726 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-89bdz"] Sep 09 23:11:59.015978 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:11:59.015954 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0201c758_08a3_4b63_ae0b_820e3a0577ca.slice/crio-36fb9cbdb7425d8104f8b32c3b56d5cef6676edeeb82cbd5abb25819e7cfbc44 WatchSource:0}: Error finding container 36fb9cbdb7425d8104f8b32c3b56d5cef6676edeeb82cbd5abb25819e7cfbc44: Status 404 returned error can't find the container with id 36fb9cbdb7425d8104f8b32c3b56d5cef6676edeeb82cbd5abb25819e7cfbc44 Sep 09 23:11:59.915649 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:11:59.915609 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-89bdz" event={"ID":"0201c758-08a3-4b63-ae0b-820e3a0577ca","Type":"ContainerStarted","Data":"36fb9cbdb7425d8104f8b32c3b56d5cef6676edeeb82cbd5abb25819e7cfbc44"} Sep 09 23:12:03.931185 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:03.931161 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 09 23:12:04.944920 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:04.944887 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-89bdz" event={"ID":"0201c758-08a3-4b63-ae0b-820e3a0577ca","Type":"ContainerStarted","Data":"e0907e78cecd935284a173105b32caedbaafed0012643cf2cf7ef46f6ab1de5a"} Sep 09 23:12:04.945462 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:04.945013 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-89bdz" Sep 09 23:12:04.960303 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:04.960254 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-89bdz" podStartSLOduration=2.04888547 podStartE2EDuration="6.960238796s" totalImagesPullingTime="4.911353326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:11:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:12:04.959036701 +0000 UTC m=+785.709621642" watchObservedRunningTime="2026-09-09 23:12:04.960238796 +0000 UTC m=+785.710823734" Sep 09 23:12:10.980438 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:10.980409 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-89bdz" Sep 09 23:12:11.775345 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:11.775313 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-69b866444b-jgwxv"] Sep 09 23:12:11.783261 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:11.783234 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69b866444b-jgwxv"] Sep 09 23:12:11.783437 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:11.783338 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69b866444b-jgwxv" Sep 09 23:12:11.872321 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:11.872286 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmffk\" (UniqueName: \"kubernetes.io/projected/13a9b7a5-bc44-4b3d-b1aa-defe9062d9fb-kube-api-access-qmffk\") pod \"authorino-69b866444b-jgwxv\" (UID: \"13a9b7a5-bc44-4b3d-b1aa-defe9062d9fb\") " pod="kuadrant-system/authorino-69b866444b-jgwxv" Sep 09 23:12:11.972812 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:11.972780 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmffk\" (UniqueName: \"kubernetes.io/projected/13a9b7a5-bc44-4b3d-b1aa-defe9062d9fb-kube-api-access-qmffk\") pod \"authorino-69b866444b-jgwxv\" (UID: \"13a9b7a5-bc44-4b3d-b1aa-defe9062d9fb\") " pod="kuadrant-system/authorino-69b866444b-jgwxv" Sep 09 23:12:11.981514 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:11.981487 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmffk\" (UniqueName: \"kubernetes.io/projected/13a9b7a5-bc44-4b3d-b1aa-defe9062d9fb-kube-api-access-qmffk\") pod \"authorino-69b866444b-jgwxv\" (UID: \"13a9b7a5-bc44-4b3d-b1aa-defe9062d9fb\") " pod="kuadrant-system/authorino-69b866444b-jgwxv" Sep 09 23:12:12.093766 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:12.093687 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69b866444b-jgwxv" Sep 09 23:12:12.362166 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:12.362089 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69b866444b-jgwxv"] Sep 09 23:12:12.365483 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:12:12.365457 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13a9b7a5_bc44_4b3d_b1aa_defe9062d9fb.slice/crio-7944c4e068717b968380eeb4761693a0ff2dde043d3d0af679c78da8f89e8978 WatchSource:0}: Error finding container 7944c4e068717b968380eeb4761693a0ff2dde043d3d0af679c78da8f89e8978: Status 404 returned error can't find the container with id 7944c4e068717b968380eeb4761693a0ff2dde043d3d0af679c78da8f89e8978 Sep 09 23:12:12.987154 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:12.987118 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69b866444b-jgwxv" event={"ID":"13a9b7a5-bc44-4b3d-b1aa-defe9062d9fb","Type":"ContainerStarted","Data":"01141835ce0514f1048baa2eae62fd2eb75df913ec12e40bb82e7be61f4d9145"} Sep 09 23:12:12.987154 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:12.987162 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69b866444b-jgwxv" event={"ID":"13a9b7a5-bc44-4b3d-b1aa-defe9062d9fb","Type":"ContainerStarted","Data":"7944c4e068717b968380eeb4761693a0ff2dde043d3d0af679c78da8f89e8978"} Sep 09 23:12:13.001864 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:13.001814 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-69b866444b-jgwxv" podStartSLOduration=1.594884248 podStartE2EDuration="2.001798955s" totalImagesPullingTime="406.914707ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:12:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:12:12.999852575 +0000 UTC m=+793.750437503" watchObservedRunningTime="2026-09-09 23:12:13.001798955 +0000 UTC m=+793.752383882" Sep 09 23:12:13.025124 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:13.025051 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wsqqd"] Sep 09 23:12:13.025281 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:13.025241 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-wsqqd" podUID="c032dd92-a7c4-4416-a4c5-a0e839650dcd" containerName="authorino" containerID="cri-o://b3ca9aa0545aa60ddd7bc4deb7ee8d88fc96911c2279fea1da5bb50f7da2a65e" gracePeriod=30 Sep 09 23:12:13.102933 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:13.102901 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6b97c6dc7b-dz86v"] Sep 09 23:12:13.106589 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:13.106567 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b97c6dc7b-dz86v" Sep 09 23:12:13.108864 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:13.108843 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 09 23:12:13.112928 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:13.112903 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b97c6dc7b-dz86v"] Sep 09 23:12:13.183346 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:13.183315 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a482aa99-74a9-42e6-bfb3-bac5e5f6ca88-tls-cert\") pod \"authorino-6b97c6dc7b-dz86v\" (UID: \"a482aa99-74a9-42e6-bfb3-bac5e5f6ca88\") " pod="kuadrant-system/authorino-6b97c6dc7b-dz86v" Sep 09 23:12:13.183528 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:13.183422 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6vrk\" (UniqueName: \"kubernetes.io/projected/a482aa99-74a9-42e6-bfb3-bac5e5f6ca88-kube-api-access-d6vrk\") pod \"authorino-6b97c6dc7b-dz86v\" (UID: \"a482aa99-74a9-42e6-bfb3-bac5e5f6ca88\") " pod="kuadrant-system/authorino-6b97c6dc7b-dz86v" Sep 09 23:12:13.281388 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:13.281316 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-wsqqd" Sep 09 23:12:13.284200 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:13.284180 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6vrk\" (UniqueName: \"kubernetes.io/projected/a482aa99-74a9-42e6-bfb3-bac5e5f6ca88-kube-api-access-d6vrk\") pod \"authorino-6b97c6dc7b-dz86v\" (UID: \"a482aa99-74a9-42e6-bfb3-bac5e5f6ca88\") " pod="kuadrant-system/authorino-6b97c6dc7b-dz86v" Sep 09 23:12:13.284297 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:13.284218 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a482aa99-74a9-42e6-bfb3-bac5e5f6ca88-tls-cert\") pod \"authorino-6b97c6dc7b-dz86v\" (UID: \"a482aa99-74a9-42e6-bfb3-bac5e5f6ca88\") " pod="kuadrant-system/authorino-6b97c6dc7b-dz86v" Sep 09 23:12:13.286642 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:13.286624 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a482aa99-74a9-42e6-bfb3-bac5e5f6ca88-tls-cert\") pod \"authorino-6b97c6dc7b-dz86v\" (UID: \"a482aa99-74a9-42e6-bfb3-bac5e5f6ca88\") " pod="kuadrant-system/authorino-6b97c6dc7b-dz86v" Sep 09 23:12:13.295916 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:13.295891 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6vrk\" (UniqueName: \"kubernetes.io/projected/a482aa99-74a9-42e6-bfb3-bac5e5f6ca88-kube-api-access-d6vrk\") pod \"authorino-6b97c6dc7b-dz86v\" (UID: \"a482aa99-74a9-42e6-bfb3-bac5e5f6ca88\") " pod="kuadrant-system/authorino-6b97c6dc7b-dz86v" Sep 09 23:12:13.385004 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:13.384968 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnsxg\" (UniqueName: \"kubernetes.io/projected/c032dd92-a7c4-4416-a4c5-a0e839650dcd-kube-api-access-gnsxg\") pod \"c032dd92-a7c4-4416-a4c5-a0e839650dcd\" (UID: \"c032dd92-a7c4-4416-a4c5-a0e839650dcd\") " Sep 09 23:12:13.387207 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:13.387182 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c032dd92-a7c4-4416-a4c5-a0e839650dcd-kube-api-access-gnsxg" (OuterVolumeSpecName: "kube-api-access-gnsxg") pod "c032dd92-a7c4-4416-a4c5-a0e839650dcd" (UID: "c032dd92-a7c4-4416-a4c5-a0e839650dcd"). InnerVolumeSpecName "kube-api-access-gnsxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:12:13.417160 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:13.417133 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b97c6dc7b-dz86v" Sep 09 23:12:13.486131 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:13.486097 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnsxg\" (UniqueName: \"kubernetes.io/projected/c032dd92-a7c4-4416-a4c5-a0e839650dcd-kube-api-access-gnsxg\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:12:13.688490 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:13.688398 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b97c6dc7b-dz86v"] Sep 09 23:12:13.691257 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:12:13.691234 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda482aa99_74a9_42e6_bfb3_bac5e5f6ca88.slice/crio-59b4dd7af885557759a9e7e75b2cac3bb96863bd9fc0d79d14e1c7b93e13680a WatchSource:0}: Error finding container 59b4dd7af885557759a9e7e75b2cac3bb96863bd9fc0d79d14e1c7b93e13680a: Status 404 returned error can't find the container with id 59b4dd7af885557759a9e7e75b2cac3bb96863bd9fc0d79d14e1c7b93e13680a Sep 09 23:12:13.993754 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:13.993665 2578 generic.go:356] "Generic (PLEG): container finished" podID="c032dd92-a7c4-4416-a4c5-a0e839650dcd" containerID="b3ca9aa0545aa60ddd7bc4deb7ee8d88fc96911c2279fea1da5bb50f7da2a65e" exitCode=0 Sep 09 23:12:13.993754 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:13.993721 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-wsqqd" Sep 09 23:12:13.994236 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:13.993753 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-wsqqd" event={"ID":"c032dd92-a7c4-4416-a4c5-a0e839650dcd","Type":"ContainerDied","Data":"b3ca9aa0545aa60ddd7bc4deb7ee8d88fc96911c2279fea1da5bb50f7da2a65e"} Sep 09 23:12:13.994236 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:13.993802 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-wsqqd" event={"ID":"c032dd92-a7c4-4416-a4c5-a0e839650dcd","Type":"ContainerDied","Data":"64cc86e207a85e31c7fa4b66ccdabf7d8786ba10b486dd61c2e878ad8f765aee"} Sep 09 23:12:13.994236 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:13.993824 2578 scope.go:117] "RemoveContainer" containerID="b3ca9aa0545aa60ddd7bc4deb7ee8d88fc96911c2279fea1da5bb50f7da2a65e" Sep 09 23:12:13.997942 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:13.997905 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b97c6dc7b-dz86v" event={"ID":"a482aa99-74a9-42e6-bfb3-bac5e5f6ca88","Type":"ContainerStarted","Data":"59b4dd7af885557759a9e7e75b2cac3bb96863bd9fc0d79d14e1c7b93e13680a"} Sep 09 23:12:14.005000 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:14.004980 2578 scope.go:117] "RemoveContainer" containerID="b3ca9aa0545aa60ddd7bc4deb7ee8d88fc96911c2279fea1da5bb50f7da2a65e" Sep 09 23:12:14.005266 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:12:14.005248 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3ca9aa0545aa60ddd7bc4deb7ee8d88fc96911c2279fea1da5bb50f7da2a65e\": container with ID starting with b3ca9aa0545aa60ddd7bc4deb7ee8d88fc96911c2279fea1da5bb50f7da2a65e not found: ID does not exist" containerID="b3ca9aa0545aa60ddd7bc4deb7ee8d88fc96911c2279fea1da5bb50f7da2a65e" Sep 09 23:12:14.005321 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:14.005280 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3ca9aa0545aa60ddd7bc4deb7ee8d88fc96911c2279fea1da5bb50f7da2a65e"} err="failed to get container status \"b3ca9aa0545aa60ddd7bc4deb7ee8d88fc96911c2279fea1da5bb50f7da2a65e\": rpc error: code = NotFound desc = could not find container \"b3ca9aa0545aa60ddd7bc4deb7ee8d88fc96911c2279fea1da5bb50f7da2a65e\": container with ID starting with b3ca9aa0545aa60ddd7bc4deb7ee8d88fc96911c2279fea1da5bb50f7da2a65e not found: ID does not exist" Sep 09 23:12:14.010681 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:14.010659 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wsqqd"] Sep 09 23:12:14.014791 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:14.014772 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wsqqd"] Sep 09 23:12:15.005501 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:15.005468 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b97c6dc7b-dz86v" event={"ID":"a482aa99-74a9-42e6-bfb3-bac5e5f6ca88","Type":"ContainerStarted","Data":"7dd327432c54a636fb1c563fc8d2114e894ed8efc7f05ddeb164d16356d7da43"} Sep 09 23:12:15.020707 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:15.020652 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6b97c6dc7b-dz86v" podStartSLOduration=1.4812042029999999 podStartE2EDuration="2.020636653s" totalImagesPullingTime="539.43245ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:12:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:12:15.019108738 +0000 UTC m=+795.769693664" watchObservedRunningTime="2026-09-09 23:12:15.020636653 +0000 UTC m=+795.771221579" Sep 09 23:12:15.047514 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:15.047474 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69b866444b-jgwxv"] Sep 09 23:12:15.047681 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:15.047653 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-69b866444b-jgwxv" podUID="13a9b7a5-bc44-4b3d-b1aa-defe9062d9fb" containerName="authorino" containerID="cri-o://01141835ce0514f1048baa2eae62fd2eb75df913ec12e40bb82e7be61f4d9145" gracePeriod=30 Sep 09 23:12:15.302645 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:15.302621 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69b866444b-jgwxv" Sep 09 23:12:15.402537 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:15.402502 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmffk\" (UniqueName: \"kubernetes.io/projected/13a9b7a5-bc44-4b3d-b1aa-defe9062d9fb-kube-api-access-qmffk\") pod \"13a9b7a5-bc44-4b3d-b1aa-defe9062d9fb\" (UID: \"13a9b7a5-bc44-4b3d-b1aa-defe9062d9fb\") " Sep 09 23:12:15.404793 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:15.404766 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a9b7a5-bc44-4b3d-b1aa-defe9062d9fb-kube-api-access-qmffk" (OuterVolumeSpecName: "kube-api-access-qmffk") pod "13a9b7a5-bc44-4b3d-b1aa-defe9062d9fb" (UID: "13a9b7a5-bc44-4b3d-b1aa-defe9062d9fb"). InnerVolumeSpecName "kube-api-access-qmffk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:12:15.504047 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:15.504010 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmffk\" (UniqueName: \"kubernetes.io/projected/13a9b7a5-bc44-4b3d-b1aa-defe9062d9fb-kube-api-access-qmffk\") on node \"ip-10-0-139-252.ec2.internal\" DevicePath \"\"" Sep 09 23:12:15.851880 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:15.851839 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c032dd92-a7c4-4416-a4c5-a0e839650dcd" path="/var/lib/kubelet/pods/c032dd92-a7c4-4416-a4c5-a0e839650dcd/volumes" Sep 09 23:12:16.012274 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:16.012241 2578 generic.go:356] "Generic (PLEG): container finished" podID="13a9b7a5-bc44-4b3d-b1aa-defe9062d9fb" containerID="01141835ce0514f1048baa2eae62fd2eb75df913ec12e40bb82e7be61f4d9145" exitCode=0 Sep 09 23:12:16.012727 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:16.012286 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69b866444b-jgwxv" Sep 09 23:12:16.012727 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:16.012336 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69b866444b-jgwxv" event={"ID":"13a9b7a5-bc44-4b3d-b1aa-defe9062d9fb","Type":"ContainerDied","Data":"01141835ce0514f1048baa2eae62fd2eb75df913ec12e40bb82e7be61f4d9145"} Sep 09 23:12:16.012727 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:16.012374 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69b866444b-jgwxv" event={"ID":"13a9b7a5-bc44-4b3d-b1aa-defe9062d9fb","Type":"ContainerDied","Data":"7944c4e068717b968380eeb4761693a0ff2dde043d3d0af679c78da8f89e8978"} Sep 09 23:12:16.012727 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:16.012418 2578 scope.go:117] "RemoveContainer" containerID="01141835ce0514f1048baa2eae62fd2eb75df913ec12e40bb82e7be61f4d9145" Sep 09 23:12:16.021010 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:16.020981 2578 scope.go:117] "RemoveContainer" containerID="01141835ce0514f1048baa2eae62fd2eb75df913ec12e40bb82e7be61f4d9145" Sep 09 23:12:16.021281 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:12:16.021262 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01141835ce0514f1048baa2eae62fd2eb75df913ec12e40bb82e7be61f4d9145\": container with ID starting with 01141835ce0514f1048baa2eae62fd2eb75df913ec12e40bb82e7be61f4d9145 not found: ID does not exist" containerID="01141835ce0514f1048baa2eae62fd2eb75df913ec12e40bb82e7be61f4d9145" Sep 09 23:12:16.021334 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:16.021291 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01141835ce0514f1048baa2eae62fd2eb75df913ec12e40bb82e7be61f4d9145"} err="failed to get container status \"01141835ce0514f1048baa2eae62fd2eb75df913ec12e40bb82e7be61f4d9145\": rpc error: code = NotFound desc = could not find container \"01141835ce0514f1048baa2eae62fd2eb75df913ec12e40bb82e7be61f4d9145\": container with ID starting with 01141835ce0514f1048baa2eae62fd2eb75df913ec12e40bb82e7be61f4d9145 not found: ID does not exist" Sep 09 23:12:16.029576 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:16.029553 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69b866444b-jgwxv"] Sep 09 23:12:16.031287 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:16.031262 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-69b866444b-jgwxv"] Sep 09 23:12:17.852340 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:17.852297 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13a9b7a5-bc44-4b3d-b1aa-defe9062d9fb" path="/var/lib/kubelet/pods/13a9b7a5-bc44-4b3d-b1aa-defe9062d9fb/volumes" Sep 09 23:12:53.636177 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.636141 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-79k7l"] Sep 09 23:12:53.636623 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.636570 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c032dd92-a7c4-4416-a4c5-a0e839650dcd" containerName="authorino" Sep 09 23:12:53.636623 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.636584 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c032dd92-a7c4-4416-a4c5-a0e839650dcd" containerName="authorino" Sep 09 23:12:53.636623 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.636606 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13a9b7a5-bc44-4b3d-b1aa-defe9062d9fb" containerName="authorino" Sep 09 23:12:53.636623 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.636611 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="13a9b7a5-bc44-4b3d-b1aa-defe9062d9fb" containerName="authorino" Sep 09 23:12:53.636749 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.636669 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c032dd92-a7c4-4416-a4c5-a0e839650dcd" containerName="authorino" Sep 09 23:12:53.636749 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.636679 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="13a9b7a5-bc44-4b3d-b1aa-defe9062d9fb" containerName="authorino" Sep 09 23:12:53.654348 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.654319 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-79k7l"] Sep 09 23:12:53.654515 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.654457 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-79k7l" Sep 09 23:12:53.656821 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.656794 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5kj72\"" Sep 09 23:12:53.656951 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.656830 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 09 23:12:53.663067 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.663045 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-wd9cw"] Sep 09 23:12:53.665409 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.665364 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-wd9cw" Sep 09 23:12:53.680842 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.680820 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-wd9cw"] Sep 09 23:12:53.713345 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.713310 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f6be58-73a2-4599-87f6-b125b9a46d0d-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-79k7l\" (UID: \"70f6be58-73a2-4599-87f6-b125b9a46d0d\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-79k7l" Sep 09 23:12:53.713524 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.713356 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43afd992-8822-4f8b-a80e-02a5ffd0fc0a-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-wd9cw\" (UID: \"43afd992-8822-4f8b-a80e-02a5ffd0fc0a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-wd9cw" Sep 09 23:12:53.713524 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.713470 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjsr9\" (UniqueName: \"kubernetes.io/projected/43afd992-8822-4f8b-a80e-02a5ffd0fc0a-kube-api-access-kjsr9\") pod \"payload-processing-5b54b7fbb7-wd9cw\" (UID: \"43afd992-8822-4f8b-a80e-02a5ffd0fc0a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-wd9cw" Sep 09 23:12:53.814782 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.814749 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f6be58-73a2-4599-87f6-b125b9a46d0d-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-79k7l\" (UID: \"70f6be58-73a2-4599-87f6-b125b9a46d0d\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-79k7l" Sep 09 23:12:53.814782 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.814786 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43afd992-8822-4f8b-a80e-02a5ffd0fc0a-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-wd9cw\" (UID: \"43afd992-8822-4f8b-a80e-02a5ffd0fc0a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-wd9cw" Sep 09 23:12:53.815040 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.814811 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjsr9\" (UniqueName: \"kubernetes.io/projected/43afd992-8822-4f8b-a80e-02a5ffd0fc0a-kube-api-access-kjsr9\") pod \"payload-processing-5b54b7fbb7-wd9cw\" (UID: \"43afd992-8822-4f8b-a80e-02a5ffd0fc0a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-wd9cw" Sep 09 23:12:53.815040 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.814864 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8twm\" (UniqueName: \"kubernetes.io/projected/70f6be58-73a2-4599-87f6-b125b9a46d0d-kube-api-access-g8twm\") pod \"payload-pre-processing-59fbc47bf8-79k7l\" (UID: \"70f6be58-73a2-4599-87f6-b125b9a46d0d\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-79k7l" Sep 09 23:12:53.815529 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.815505 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f6be58-73a2-4599-87f6-b125b9a46d0d-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-79k7l\" (UID: \"70f6be58-73a2-4599-87f6-b125b9a46d0d\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-79k7l" Sep 09 23:12:53.815628 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.815547 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43afd992-8822-4f8b-a80e-02a5ffd0fc0a-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-wd9cw\" (UID: \"43afd992-8822-4f8b-a80e-02a5ffd0fc0a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-wd9cw" Sep 09 23:12:53.823014 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.822992 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjsr9\" (UniqueName: \"kubernetes.io/projected/43afd992-8822-4f8b-a80e-02a5ffd0fc0a-kube-api-access-kjsr9\") pod \"payload-processing-5b54b7fbb7-wd9cw\" (UID: \"43afd992-8822-4f8b-a80e-02a5ffd0fc0a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-wd9cw" Sep 09 23:12:53.915913 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.915814 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8twm\" (UniqueName: \"kubernetes.io/projected/70f6be58-73a2-4599-87f6-b125b9a46d0d-kube-api-access-g8twm\") pod \"payload-pre-processing-59fbc47bf8-79k7l\" (UID: \"70f6be58-73a2-4599-87f6-b125b9a46d0d\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-79k7l" Sep 09 23:12:53.927584 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.927551 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8twm\" (UniqueName: \"kubernetes.io/projected/70f6be58-73a2-4599-87f6-b125b9a46d0d-kube-api-access-g8twm\") pod \"payload-pre-processing-59fbc47bf8-79k7l\" (UID: \"70f6be58-73a2-4599-87f6-b125b9a46d0d\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-79k7l" Sep 09 23:12:53.966303 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.966236 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-79k7l" Sep 09 23:12:53.975976 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:53.975949 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-wd9cw" Sep 09 23:12:54.251674 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:54.251642 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-79k7l"] Sep 09 23:12:54.259167 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:54.259127 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-wd9cw"] Sep 09 23:12:55.219073 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:55.219034 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-79k7l" event={"ID":"70f6be58-73a2-4599-87f6-b125b9a46d0d","Type":"ContainerStarted","Data":"a11997bbdbadf62bc49ecc969460f8bfa7db408db42bb81d3ba2076fa14c5286"} Sep 09 23:12:55.221232 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:55.221124 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-wd9cw" event={"ID":"43afd992-8822-4f8b-a80e-02a5ffd0fc0a","Type":"ContainerStarted","Data":"dd8e30d12e807dd883851baf618c71e37507154680cfc74bb8fecd0bd73a80d6"} Sep 09 23:12:57.234616 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:57.234573 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-79k7l" event={"ID":"70f6be58-73a2-4599-87f6-b125b9a46d0d","Type":"ContainerStarted","Data":"b5ac86c071bfae41fc712fdba5b034225e4ea4db25091a4522e3ad55b23a84c6"} Sep 09 23:12:57.235472 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:57.235445 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-79k7l" Sep 09 23:12:57.237683 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:57.237654 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-wd9cw" event={"ID":"43afd992-8822-4f8b-a80e-02a5ffd0fc0a","Type":"ContainerStarted","Data":"ee8b33e0146ae2481b95cca55cea5ad40fdec06cac3f4b4cacd809e353c35ee1"} Sep 09 23:12:57.238181 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:57.238158 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-wd9cw" Sep 09 23:12:57.255953 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:57.255888 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-79k7l" podStartSLOduration=1.935312769 podStartE2EDuration="4.255869102s" totalImagesPullingTime="2.320556333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:12:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:12:57.253487238 +0000 UTC m=+838.004072168" watchObservedRunningTime="2026-09-09 23:12:57.255869102 +0000 UTC m=+838.006454030" Sep 09 23:12:57.272790 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:12:57.272735 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-wd9cw" podStartSLOduration=1.95834494 podStartE2EDuration="4.27271966s" totalImagesPullingTime="2.31437472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:12:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:12:57.271959692 +0000 UTC m=+838.022544641" watchObservedRunningTime="2026-09-09 23:12:57.27271966 +0000 UTC m=+838.023304587" Sep 09 23:13:04.284458 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:04.284417 2578 generic.go:356] "Generic (PLEG): container finished" podID="1a051327-0bb5-4299-a04e-b402de10abf8" containerID="16d830779fe27fa7cba737eb91a738268c7bc02e65b8ec0b925d3ade3a5c4507" exitCode=0 Sep 09 23:13:04.285080 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:04.284480 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" event={"ID":"1a051327-0bb5-4299-a04e-b402de10abf8","Type":"ContainerDied","Data":"16d830779fe27fa7cba737eb91a738268c7bc02e65b8ec0b925d3ade3a5c4507"} Sep 09 23:13:04.285080 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:04.284858 2578 scope.go:117] "RemoveContainer" containerID="16d830779fe27fa7cba737eb91a738268c7bc02e65b8ec0b925d3ade3a5c4507" Sep 09 23:13:04.285890 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:04.285862 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 09 23:13:04.285983 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:04.285934 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 09 23:13:04.285983 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:04.285970 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 09 23:13:05.291148 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:05.291110 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" event={"ID":"1a051327-0bb5-4299-a04e-b402de10abf8","Type":"ContainerStarted","Data":"4c581bdfeb1a484d305836f67e00a39c2283b994b5f9637e0369652eb101a972"} Sep 09 23:13:05.323260 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:05.323234 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:13:05.324511 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:05.324482 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lzj2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 09 23:13:05.324619 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:05.324527 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" podUID="1a051327-0bb5-4299-a04e-b402de10abf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 09 23:13:06.323325 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:06.323286 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lzj2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 09 23:13:06.323727 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:06.323343 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" podUID="1a051327-0bb5-4299-a04e-b402de10abf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 09 23:13:06.773195 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:06.773161 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:13:07.323749 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:07.323718 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lzj2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 09 23:13:07.324128 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:07.323778 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" podUID="1a051327-0bb5-4299-a04e-b402de10abf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 09 23:13:08.323401 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:08.323347 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lzj2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 09 23:13:08.323574 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:08.323426 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" podUID="1a051327-0bb5-4299-a04e-b402de10abf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 09 23:13:09.257476 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:09.256576 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-wd9cw" Sep 09 23:13:09.258781 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:09.258089 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-79k7l" Sep 09 23:13:09.323592 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:09.323557 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lzj2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 09 23:13:09.323743 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:09.323621 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" podUID="1a051327-0bb5-4299-a04e-b402de10abf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 09 23:13:10.326866 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:10.326838 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:13:10.327790 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:10.327772 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lzj2s" Sep 09 23:13:41.994142 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:41.994110 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn"] Sep 09 23:13:41.997094 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:41.997075 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" Sep 09 23:13:42.001362 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:42.001338 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cl4wn\"" Sep 09 23:13:42.002541 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:42.002512 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 09 23:13:42.002664 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:42.002581 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 09 23:13:42.002664 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:42.002512 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 09 23:13:42.012460 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:42.012435 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn"] Sep 09 23:13:42.089903 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:42.089865 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn\" (UID: \"87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" Sep 09 23:13:42.090076 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:42.089922 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn\" (UID: \"87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" Sep 09 23:13:42.090076 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:42.089990 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn\" (UID: \"87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" Sep 09 23:13:42.090076 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:42.090040 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hpk4\" (UniqueName: \"kubernetes.io/projected/87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b-kube-api-access-2hpk4\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn\" (UID: \"87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" Sep 09 23:13:42.090206 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:42.090086 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn\" (UID: \"87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" Sep 09 23:13:42.090206 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:42.090177 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn\" (UID: \"87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" Sep 09 23:13:42.190709 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:42.190676 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn\" (UID: \"87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" Sep 09 23:13:42.190709 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:42.190711 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn\" (UID: \"87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" Sep 09 23:13:42.190925 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:42.190735 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hpk4\" (UniqueName: \"kubernetes.io/projected/87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b-kube-api-access-2hpk4\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn\" (UID: \"87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" Sep 09 23:13:42.190925 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:42.190870 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn\" (UID: \"87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" Sep 09 23:13:42.190992 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:42.190967 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn\" (UID: \"87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" Sep 09 23:13:42.191026 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:42.191005 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn\" (UID: \"87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" Sep 09 23:13:42.191182 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:42.191164 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn\" (UID: \"87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" Sep 09 23:13:42.191278 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:42.191258 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn\" (UID: \"87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" Sep 09 23:13:42.191329 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:42.191285 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn\" (UID: \"87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" Sep 09 23:13:42.193122 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:42.193097 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn\" (UID: \"87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" Sep 09 23:13:42.193578 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:42.193557 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn\" (UID: \"87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" Sep 09 23:13:42.200473 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:42.200451 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hpk4\" (UniqueName: \"kubernetes.io/projected/87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b-kube-api-access-2hpk4\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn\" (UID: \"87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" Sep 09 23:13:42.307268 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:42.307235 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" Sep 09 23:13:42.586408 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:42.586365 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn"] Sep 09 23:13:42.588274 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:13:42.588243 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87779c0d_4efe_443a_a8ce_5c1bf7b0ae1b.slice/crio-81959111afecb2c139304069ad0b668314b70815b0682774c0bb02b0da921c9c WatchSource:0}: Error finding container 81959111afecb2c139304069ad0b668314b70815b0682774c0bb02b0da921c9c: Status 404 returned error can't find the container with id 81959111afecb2c139304069ad0b668314b70815b0682774c0bb02b0da921c9c Sep 09 23:13:43.499138 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:43.499094 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" event={"ID":"87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b","Type":"ContainerStarted","Data":"81959111afecb2c139304069ad0b668314b70815b0682774c0bb02b0da921c9c"} Sep 09 23:13:44.506893 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:44.506856 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" event={"ID":"87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b","Type":"ContainerStarted","Data":"27e1b8c468cea37ae8bbb9c746f668d2adfe13fbca8f8426d6d8ad3d32cae68e"} Sep 09 23:13:44.525056 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:44.525011 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" podStartSLOduration=2.5450571269999998 podStartE2EDuration="3.5249964s" totalImagesPullingTime="979.939273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:13:44.523466352 +0000 UTC m=+885.274051299" watchObservedRunningTime="2026-09-09 23:13:44.5249964 +0000 UTC m=+885.275581327" Sep 09 23:13:52.307450 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:52.307334 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" Sep 09 23:13:52.308016 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:52.307460 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" Sep 09 23:13:52.319795 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:52.319771 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" Sep 09 23:13:52.560286 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:13:52.560201 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn" Sep 09 23:14:04.203241 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:04.203204 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg"] Sep 09 23:14:04.207476 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:04.207456 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:04.209706 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:04.209685 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 09 23:14:04.214911 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:04.214889 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg"] Sep 09 23:14:04.300470 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:04.300443 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cbd14d2-0fc0-4d69-a7ca-001692735fd9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg\" (UID: \"5cbd14d2-0fc0-4d69-a7ca-001692735fd9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:04.300627 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:04.300549 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtbzp\" (UniqueName: \"kubernetes.io/projected/5cbd14d2-0fc0-4d69-a7ca-001692735fd9-kube-api-access-rtbzp\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg\" (UID: \"5cbd14d2-0fc0-4d69-a7ca-001692735fd9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:04.300627 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:04.300594 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbd14d2-0fc0-4d69-a7ca-001692735fd9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg\" (UID: \"5cbd14d2-0fc0-4d69-a7ca-001692735fd9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:04.300627 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:04.300618 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cbd14d2-0fc0-4d69-a7ca-001692735fd9-home\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg\" (UID: \"5cbd14d2-0fc0-4d69-a7ca-001692735fd9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:04.300734 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:04.300647 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cbd14d2-0fc0-4d69-a7ca-001692735fd9-dshm\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg\" (UID: \"5cbd14d2-0fc0-4d69-a7ca-001692735fd9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:04.300769 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:04.300738 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cbd14d2-0fc0-4d69-a7ca-001692735fd9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg\" (UID: \"5cbd14d2-0fc0-4d69-a7ca-001692735fd9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:04.401465 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:04.401431 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbd14d2-0fc0-4d69-a7ca-001692735fd9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg\" (UID: \"5cbd14d2-0fc0-4d69-a7ca-001692735fd9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:04.401628 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:04.401475 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cbd14d2-0fc0-4d69-a7ca-001692735fd9-home\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg\" (UID: \"5cbd14d2-0fc0-4d69-a7ca-001692735fd9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:04.401628 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:04.401507 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cbd14d2-0fc0-4d69-a7ca-001692735fd9-dshm\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg\" (UID: \"5cbd14d2-0fc0-4d69-a7ca-001692735fd9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:04.401628 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:04.401617 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cbd14d2-0fc0-4d69-a7ca-001692735fd9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg\" (UID: \"5cbd14d2-0fc0-4d69-a7ca-001692735fd9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:04.401781 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:04.401657 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cbd14d2-0fc0-4d69-a7ca-001692735fd9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg\" (UID: \"5cbd14d2-0fc0-4d69-a7ca-001692735fd9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:04.401781 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:04.401767 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtbzp\" (UniqueName: \"kubernetes.io/projected/5cbd14d2-0fc0-4d69-a7ca-001692735fd9-kube-api-access-rtbzp\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg\" (UID: \"5cbd14d2-0fc0-4d69-a7ca-001692735fd9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:04.401906 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:04.401884 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cbd14d2-0fc0-4d69-a7ca-001692735fd9-home\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg\" (UID: \"5cbd14d2-0fc0-4d69-a7ca-001692735fd9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:04.401973 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:04.401915 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cbd14d2-0fc0-4d69-a7ca-001692735fd9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg\" (UID: \"5cbd14d2-0fc0-4d69-a7ca-001692735fd9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:04.402168 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:04.402151 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cbd14d2-0fc0-4d69-a7ca-001692735fd9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg\" (UID: \"5cbd14d2-0fc0-4d69-a7ca-001692735fd9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:04.403953 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:04.403916 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cbd14d2-0fc0-4d69-a7ca-001692735fd9-dshm\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg\" (UID: \"5cbd14d2-0fc0-4d69-a7ca-001692735fd9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:04.404078 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:04.404047 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbd14d2-0fc0-4d69-a7ca-001692735fd9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg\" (UID: \"5cbd14d2-0fc0-4d69-a7ca-001692735fd9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:04.410773 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:04.410750 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtbzp\" (UniqueName: \"kubernetes.io/projected/5cbd14d2-0fc0-4d69-a7ca-001692735fd9-kube-api-access-rtbzp\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg\" (UID: \"5cbd14d2-0fc0-4d69-a7ca-001692735fd9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:04.518679 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:04.518651 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:04.794826 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:04.794804 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg"] Sep 09 23:14:04.797241 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:14:04.797217 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cbd14d2_0fc0_4d69_a7ca_001692735fd9.slice/crio-a9e5200dc617bb38df11edd5c2822a7eaa2a9aa2f0246b21cc5fb6c7d87794a6 WatchSource:0}: Error finding container a9e5200dc617bb38df11edd5c2822a7eaa2a9aa2f0246b21cc5fb6c7d87794a6: Status 404 returned error can't find the container with id a9e5200dc617bb38df11edd5c2822a7eaa2a9aa2f0246b21cc5fb6c7d87794a6 Sep 09 23:14:05.621736 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:05.621704 2578 generic.go:356] "Generic (PLEG): container finished" podID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" containerID="117e622ea4c351b05b3e4d06920464218ed275026e80b0ed32c354564b846a95" exitCode=0 Sep 09 23:14:05.622151 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:05.621791 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" event={"ID":"5cbd14d2-0fc0-4d69-a7ca-001692735fd9","Type":"ContainerDied","Data":"117e622ea4c351b05b3e4d06920464218ed275026e80b0ed32c354564b846a95"} Sep 09 23:14:05.622151 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:05.621849 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" event={"ID":"5cbd14d2-0fc0-4d69-a7ca-001692735fd9","Type":"ContainerStarted","Data":"a9e5200dc617bb38df11edd5c2822a7eaa2a9aa2f0246b21cc5fb6c7d87794a6"} Sep 09 23:14:05.622151 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:05.622098 2578 scope.go:117] "RemoveContainer" containerID="117e622ea4c351b05b3e4d06920464218ed275026e80b0ed32c354564b846a95" Sep 09 23:14:06.628579 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:06.628540 2578 generic.go:356] "Generic (PLEG): container finished" podID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" containerID="81d1470a799c87027f4b4645becb5a4b807ce3f4fe7a45df920a532578e540db" exitCode=0 Sep 09 23:14:06.628982 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:06.628634 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" event={"ID":"5cbd14d2-0fc0-4d69-a7ca-001692735fd9","Type":"ContainerDied","Data":"81d1470a799c87027f4b4645becb5a4b807ce3f4fe7a45df920a532578e540db"} Sep 09 23:14:06.628982 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:06.628711 2578 scope.go:117] "RemoveContainer" containerID="117e622ea4c351b05b3e4d06920464218ed275026e80b0ed32c354564b846a95" Sep 09 23:14:06.629078 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:06.628978 2578 scope.go:117] "RemoveContainer" containerID="81d1470a799c87027f4b4645becb5a4b807ce3f4fe7a45df920a532578e540db" Sep 09 23:14:06.629230 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:14:06.629198 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:14:07.635294 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:07.635265 2578 scope.go:117] "RemoveContainer" containerID="81d1470a799c87027f4b4645becb5a4b807ce3f4fe7a45df920a532578e540db" Sep 09 23:14:07.635776 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:14:07.635453 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:14:14.519099 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:14.519063 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:14.519623 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:14.519123 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:14.519623 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:14.519141 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:14.519623 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:14.519171 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:14.519623 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:14.519560 2578 scope.go:117] "RemoveContainer" containerID="81d1470a799c87027f4b4645becb5a4b807ce3f4fe7a45df920a532578e540db" Sep 09 23:14:14.519770 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:14:14.519730 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:14:27.846868 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:27.846830 2578 scope.go:117] "RemoveContainer" containerID="81d1470a799c87027f4b4645becb5a4b807ce3f4fe7a45df920a532578e540db" Sep 09 23:14:27.847255 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:27.847131 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 23:14:28.746009 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:28.745979 2578 generic.go:356] "Generic (PLEG): container finished" podID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" containerID="9a243951a1ac7cc254eedaae0e4e5def57380bcc1931e98fffe08fc8b0243a5d" exitCode=0 Sep 09 23:14:28.746187 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:28.746055 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" event={"ID":"5cbd14d2-0fc0-4d69-a7ca-001692735fd9","Type":"ContainerDied","Data":"9a243951a1ac7cc254eedaae0e4e5def57380bcc1931e98fffe08fc8b0243a5d"} Sep 09 23:14:28.746187 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:28.746116 2578 scope.go:117] "RemoveContainer" containerID="81d1470a799c87027f4b4645becb5a4b807ce3f4fe7a45df920a532578e540db" Sep 09 23:14:28.746528 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:28.746507 2578 scope.go:117] "RemoveContainer" containerID="9a243951a1ac7cc254eedaae0e4e5def57380bcc1931e98fffe08fc8b0243a5d" Sep 09 23:14:28.746741 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:14:28.746718 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:14:34.519435 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:34.519397 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:34.519825 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:34.519465 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:34.519825 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:34.519482 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:34.519825 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:34.519493 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:14:34.519940 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:34.519832 2578 scope.go:117] "RemoveContainer" containerID="9a243951a1ac7cc254eedaae0e4e5def57380bcc1931e98fffe08fc8b0243a5d" Sep 09 23:14:34.520018 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:14:34.519999 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:14:47.846943 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:14:47.846912 2578 scope.go:117] "RemoveContainer" containerID="9a243951a1ac7cc254eedaae0e4e5def57380bcc1931e98fffe08fc8b0243a5d" Sep 09 23:14:47.847476 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:14:47.847093 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:15:01.846737 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:15:01.846701 2578 scope.go:117] "RemoveContainer" containerID="9a243951a1ac7cc254eedaae0e4e5def57380bcc1931e98fffe08fc8b0243a5d" Sep 09 23:15:02.161208 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:15:02.161171 2578 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cbd14d2_0fc0_4d69_a7ca_001692735fd9.slice/crio-conmon-b1ea3d31fd7e702ec21170d96a760e71d03665bed5a00b201416081ea084bab5.scope\": RecentStats: unable to find data in memory cache]" Sep 09 23:15:02.927848 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:15:02.927813 2578 generic.go:356] "Generic (PLEG): container finished" podID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" containerID="b1ea3d31fd7e702ec21170d96a760e71d03665bed5a00b201416081ea084bab5" exitCode=0 Sep 09 23:15:02.928260 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:15:02.927875 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" event={"ID":"5cbd14d2-0fc0-4d69-a7ca-001692735fd9","Type":"ContainerDied","Data":"b1ea3d31fd7e702ec21170d96a760e71d03665bed5a00b201416081ea084bab5"} Sep 09 23:15:02.928260 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:15:02.927960 2578 scope.go:117] "RemoveContainer" containerID="9a243951a1ac7cc254eedaae0e4e5def57380bcc1931e98fffe08fc8b0243a5d" Sep 09 23:15:02.928473 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:15:02.928363 2578 scope.go:117] "RemoveContainer" containerID="b1ea3d31fd7e702ec21170d96a760e71d03665bed5a00b201416081ea084bab5" Sep 09 23:15:02.928642 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:15:02.928615 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:15:04.519231 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:15:04.519197 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:15:04.519691 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:15:04.519262 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:15:04.519691 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:15:04.519277 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:15:04.519691 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:15:04.519290 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:15:04.519691 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:15:04.519688 2578 scope.go:117] "RemoveContainer" containerID="b1ea3d31fd7e702ec21170d96a760e71d03665bed5a00b201416081ea084bab5" Sep 09 23:15:04.519881 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:15:04.519862 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:15:16.846779 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:15:16.846745 2578 scope.go:117] "RemoveContainer" containerID="b1ea3d31fd7e702ec21170d96a760e71d03665bed5a00b201416081ea084bab5" Sep 09 23:15:16.847250 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:15:16.846958 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:15:30.846742 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:15:30.846704 2578 scope.go:117] "RemoveContainer" containerID="b1ea3d31fd7e702ec21170d96a760e71d03665bed5a00b201416081ea084bab5" Sep 09 23:15:30.847171 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:15:30.846891 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:15:42.846920 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:15:42.846888 2578 scope.go:117] "RemoveContainer" containerID="b1ea3d31fd7e702ec21170d96a760e71d03665bed5a00b201416081ea084bab5" Sep 09 23:15:43.146481 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:15:43.146454 2578 generic.go:356] "Generic (PLEG): container finished" podID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" containerID="69514470f1ecffc6b419a3ea6c205486e6e6b767820eb24d529fc7e0b7718df6" exitCode=0 Sep 09 23:15:43.146600 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:15:43.146528 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" event={"ID":"5cbd14d2-0fc0-4d69-a7ca-001692735fd9","Type":"ContainerDied","Data":"69514470f1ecffc6b419a3ea6c205486e6e6b767820eb24d529fc7e0b7718df6"} Sep 09 23:15:43.146600 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:15:43.146588 2578 scope.go:117] "RemoveContainer" containerID="b1ea3d31fd7e702ec21170d96a760e71d03665bed5a00b201416081ea084bab5" Sep 09 23:15:43.147026 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:15:43.147008 2578 scope.go:117] "RemoveContainer" containerID="69514470f1ecffc6b419a3ea6c205486e6e6b767820eb24d529fc7e0b7718df6" Sep 09 23:15:43.147241 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:15:43.147222 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:15:44.519033 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:15:44.518999 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:15:44.519439 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:15:44.519058 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:15:44.519439 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:15:44.519075 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:15:44.519439 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:15:44.519087 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:15:44.519556 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:15:44.519456 2578 scope.go:117] "RemoveContainer" containerID="69514470f1ecffc6b419a3ea6c205486e6e6b767820eb24d529fc7e0b7718df6" Sep 09 23:15:44.519641 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:15:44.519621 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:15:55.846550 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:15:55.846519 2578 scope.go:117] "RemoveContainer" containerID="69514470f1ecffc6b419a3ea6c205486e6e6b767820eb24d529fc7e0b7718df6" Sep 09 23:15:55.847018 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:15:55.846701 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:16:10.846153 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:16:10.846114 2578 scope.go:117] "RemoveContainer" containerID="69514470f1ecffc6b419a3ea6c205486e6e6b767820eb24d529fc7e0b7718df6" Sep 09 23:16:10.846645 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:16:10.846362 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:16:24.846425 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:16:24.846365 2578 scope.go:117] "RemoveContainer" containerID="69514470f1ecffc6b419a3ea6c205486e6e6b767820eb24d529fc7e0b7718df6" Sep 09 23:16:24.846805 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:16:24.846583 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:16:38.846710 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:16:38.846678 2578 scope.go:117] "RemoveContainer" containerID="69514470f1ecffc6b419a3ea6c205486e6e6b767820eb24d529fc7e0b7718df6" Sep 09 23:16:38.847138 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:16:38.846908 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:16:51.846848 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:16:51.846776 2578 scope.go:117] "RemoveContainer" containerID="69514470f1ecffc6b419a3ea6c205486e6e6b767820eb24d529fc7e0b7718df6" Sep 09 23:16:51.847242 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:16:51.846976 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:17:06.846848 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:17:06.846816 2578 scope.go:117] "RemoveContainer" containerID="69514470f1ecffc6b419a3ea6c205486e6e6b767820eb24d529fc7e0b7718df6" Sep 09 23:17:07.581197 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:17:07.581164 2578 generic.go:356] "Generic (PLEG): container finished" podID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" containerID="d155d233f991938dc03aa4045d75dd8d5860de479c5960c2edb694846ae42a91" exitCode=0 Sep 09 23:17:07.581410 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:17:07.581206 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" event={"ID":"5cbd14d2-0fc0-4d69-a7ca-001692735fd9","Type":"ContainerDied","Data":"d155d233f991938dc03aa4045d75dd8d5860de479c5960c2edb694846ae42a91"} Sep 09 23:17:07.581410 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:17:07.581279 2578 scope.go:117] "RemoveContainer" containerID="69514470f1ecffc6b419a3ea6c205486e6e6b767820eb24d529fc7e0b7718df6" Sep 09 23:17:07.581712 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:17:07.581692 2578 scope.go:117] "RemoveContainer" containerID="d155d233f991938dc03aa4045d75dd8d5860de479c5960c2edb694846ae42a91" Sep 09 23:17:07.581956 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:17:07.581934 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:17:14.519454 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:17:14.519421 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:17:14.519838 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:17:14.519487 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:17:14.519838 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:17:14.519505 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:17:14.519838 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:17:14.519518 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" Sep 09 23:17:14.519958 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:17:14.519843 2578 scope.go:117] "RemoveContainer" containerID="d155d233f991938dc03aa4045d75dd8d5860de479c5960c2edb694846ae42a91" Sep 09 23:17:14.520027 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:17:14.520008 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:17:27.847144 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:17:27.847110 2578 scope.go:117] "RemoveContainer" containerID="d155d233f991938dc03aa4045d75dd8d5860de479c5960c2edb694846ae42a91" Sep 09 23:17:27.847592 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:17:27.847343 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:17:39.849823 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:17:39.849791 2578 scope.go:117] "RemoveContainer" containerID="d155d233f991938dc03aa4045d75dd8d5860de479c5960c2edb694846ae42a91" Sep 09 23:17:39.850402 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:17:39.850036 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:17:53.846698 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:17:53.846656 2578 scope.go:117] "RemoveContainer" containerID="d155d233f991938dc03aa4045d75dd8d5860de479c5960c2edb694846ae42a91" Sep 09 23:17:53.847150 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:17:53.846867 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:18:04.846151 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:04.846116 2578 scope.go:117] "RemoveContainer" containerID="d155d233f991938dc03aa4045d75dd8d5860de479c5960c2edb694846ae42a91" Sep 09 23:18:04.848393 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:18:04.846292 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:18:06.134654 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:06.134618 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_5cbd14d2-0fc0-4d69-a7ca-001692735fd9/main/5.log" Sep 09 23:18:06.249467 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:06.249442 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_5cbd14d2-0fc0-4d69-a7ca-001692735fd9/main/5.log" Sep 09 23:18:07.271838 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:07.271807 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6b97c6dc7b-dz86v_a482aa99-74a9-42e6-bfb3-bac5e5f6ca88/authorino/0.log" Sep 09 23:18:12.484913 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:12.484873 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-4sss6_329724e3-d1fb-46c9-b267-061c965cede0/manager/0.log" Sep 09 23:18:12.606061 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:12.606031 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-wzvqr_3828e5a3-af2d-4422-8fea-b9fd6fafd1ef/manager/0.log" Sep 09 23:18:12.832572 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:12.832543 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-rb4ng_94b353b9-a213-40d3-993f-ac949389e2f4/server/0.log" Sep 09 23:18:13.079199 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:13.079160 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65d8796fc4-9t9t4_df244f8e-1b94-47ef-9c41-f0ce67302a5d/manager/0.log" Sep 09 23:18:13.086695 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:13.086640 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65d8796fc4-9t9t4_df244f8e-1b94-47ef-9c41-f0ce67302a5d/copy-manifests/0.log" Sep 09 23:18:13.919442 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:13.919411 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-89bdz_0201c758-08a3-4b63-ae0b-820e3a0577ca/postgres/0.log" Sep 09 23:18:14.696979 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:14.696954 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6b97c6dc7b-dz86v_a482aa99-74a9-42e6-bfb3-bac5e5f6ca88/authorino/0.log" Sep 09 23:18:14.818337 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:14.818308 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-qxxdk_73149d7d-077d-43da-ab09-05b93da15865/manager/0.log" Sep 09 23:18:14.928315 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:14.928278 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-sp7hl_52d176e8-bec3-4248-a2eb-acb48c50e1b0/manager/0.log" Sep 09 23:18:15.166685 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:15.166655 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-lhm6j_c109c006-0720-4ca9-a160-fe4b3bb96349/manager/0.log" Sep 09 23:18:15.409038 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:15.409009 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-8k2zw_b6c996fa-1326-4b5d-bca2-102b32a3b791/manager/0.log" Sep 09 23:18:16.161541 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:16.161512 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7db58bdd4-lkdr8_5afd01d0-a070-4844-8eff-a6447e22b7d5/kube-auth-proxy/0.log" Sep 09 23:18:16.283301 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:16.283263 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lzj2s_1a051327-0bb5-4299-a04e-b402de10abf8/istio-proxy/1.log" Sep 09 23:18:16.397360 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:16.397329 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-79k7l_70f6be58-73a2-4599-87f6-b125b9a46d0d/payload-pre-processing/0.log" Sep 09 23:18:16.513467 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:16.513439 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-wd9cw_43afd992-8822-4f8b-a80e-02a5ffd0fc0a/payload-processing/0.log" Sep 09 23:18:17.207449 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:17.207406 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-cbhsn_87779c0d-4efe-443a-a8ce-5c1bf7b0ae1b/main/0.log" Sep 09 23:18:17.560100 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:17.560070 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_5cbd14d2-0fc0-4d69-a7ca-001692735fd9/main/5.log" Sep 09 23:18:17.846956 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:17.846879 2578 scope.go:117] "RemoveContainer" containerID="d155d233f991938dc03aa4045d75dd8d5860de479c5960c2edb694846ae42a91" Sep 09 23:18:17.847109 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:18:17.847090 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:18:25.798672 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:25.798593 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-b6l2s_55304a42-5933-48eb-bbbd-ed2f0d4b933b/global-pull-secret-syncer/0.log" Sep 09 23:18:25.957266 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:25.957236 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wc456_3c974c39-606b-4f5d-ae55-cd15d6393611/konnectivity-agent/0.log" Sep 09 23:18:26.005078 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:26.005050 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-252.ec2.internal_ebaf3d05ceef927b72c983f2c9e143f4/haproxy/0.log" Sep 09 23:18:30.407477 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:30.407449 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6b97c6dc7b-dz86v_a482aa99-74a9-42e6-bfb3-bac5e5f6ca88/authorino/0.log" Sep 09 23:18:30.440107 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:30.440064 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-qxxdk_73149d7d-077d-43da-ab09-05b93da15865/manager/0.log" Sep 09 23:18:30.464999 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:30.464972 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-sp7hl_52d176e8-bec3-4248-a2eb-acb48c50e1b0/manager/0.log" Sep 09 23:18:30.574977 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:30.574943 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-lhm6j_c109c006-0720-4ca9-a160-fe4b3bb96349/manager/0.log" Sep 09 23:18:30.634535 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:30.634505 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-8k2zw_b6c996fa-1326-4b5d-bca2-102b32a3b791/manager/0.log" Sep 09 23:18:32.185159 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:32.185126 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3fbde13f-9319-4ad4-acf8-a2eb971ff8b6/alertmanager/0.log" Sep 09 23:18:32.210596 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:32.210569 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3fbde13f-9319-4ad4-acf8-a2eb971ff8b6/config-reloader/0.log" Sep 09 23:18:32.232821 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:32.232791 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3fbde13f-9319-4ad4-acf8-a2eb971ff8b6/kube-rbac-proxy-web/0.log" Sep 09 23:18:32.258362 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:32.258336 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3fbde13f-9319-4ad4-acf8-a2eb971ff8b6/kube-rbac-proxy/0.log" Sep 09 23:18:32.284768 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:32.284747 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3fbde13f-9319-4ad4-acf8-a2eb971ff8b6/kube-rbac-proxy-metric/0.log" Sep 09 23:18:32.309902 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:32.309877 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3fbde13f-9319-4ad4-acf8-a2eb971ff8b6/prom-label-proxy/0.log" Sep 09 23:18:32.333512 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:32.333483 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3fbde13f-9319-4ad4-acf8-a2eb971ff8b6/init-config-reloader/0.log" Sep 09 23:18:32.376634 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:32.376609 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-9vck7_958bade9-9642-4a24-9b2d-cd313f77c312/cluster-monitoring-operator/0.log" Sep 09 23:18:32.536616 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:32.536584 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-8g6z5_6330f214-729e-404e-9b73-c907fe394743/monitoring-plugin/0.log" Sep 09 23:18:32.653742 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:32.653714 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s8wp2_60dc9d97-70e5-4683-85c0-539aa019f6ac/node-exporter/0.log" Sep 09 23:18:32.677878 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:32.677855 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s8wp2_60dc9d97-70e5-4683-85c0-539aa019f6ac/kube-rbac-proxy/0.log" Sep 09 23:18:32.714399 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:32.714358 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s8wp2_60dc9d97-70e5-4683-85c0-539aa019f6ac/init-textfile/0.log" Sep 09 23:18:32.846014 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:32.845931 2578 scope.go:117] "RemoveContainer" containerID="d155d233f991938dc03aa4045d75dd8d5860de479c5960c2edb694846ae42a91" Sep 09 23:18:32.846179 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:18:32.846148 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:18:33.230552 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:33.230475 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74d5594d54-qlf7h_2c4dbc94-e815-4e1c-ba62-bd74df448ca5/telemeter-client/0.log" Sep 09 23:18:33.258699 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:33.258677 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74d5594d54-qlf7h_2c4dbc94-e815-4e1c-ba62-bd74df448ca5/reload/0.log" Sep 09 23:18:33.292246 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:33.292218 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74d5594d54-qlf7h_2c4dbc94-e815-4e1c-ba62-bd74df448ca5/kube-rbac-proxy/0.log" Sep 09 23:18:36.166398 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:36.166341 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5vzlx/perf-node-gather-daemonset-rz8tn"] Sep 09 23:18:36.170013 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:36.169984 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5vzlx/perf-node-gather-daemonset-rz8tn" Sep 09 23:18:36.172296 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:36.172274 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5vzlx\"/\"default-dockercfg-tghjq\"" Sep 09 23:18:36.173168 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:36.173150 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5vzlx\"/\"kube-root-ca.crt\"" Sep 09 23:18:36.173296 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:36.173194 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5vzlx\"/\"openshift-service-ca.crt\"" Sep 09 23:18:36.178870 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:36.178849 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5vzlx/perf-node-gather-daemonset-rz8tn"] Sep 09 23:18:36.256160 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:36.256124 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/715f2358-eb71-4e18-9341-2d4e7e60dbdf-proc\") pod \"perf-node-gather-daemonset-rz8tn\" (UID: \"715f2358-eb71-4e18-9341-2d4e7e60dbdf\") " pod="openshift-must-gather-5vzlx/perf-node-gather-daemonset-rz8tn" Sep 09 23:18:36.256301 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:36.256167 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/715f2358-eb71-4e18-9341-2d4e7e60dbdf-podres\") pod \"perf-node-gather-daemonset-rz8tn\" (UID: \"715f2358-eb71-4e18-9341-2d4e7e60dbdf\") " pod="openshift-must-gather-5vzlx/perf-node-gather-daemonset-rz8tn" Sep 09 23:18:36.256301 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:36.256243 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/715f2358-eb71-4e18-9341-2d4e7e60dbdf-sys\") pod \"perf-node-gather-daemonset-rz8tn\" (UID: \"715f2358-eb71-4e18-9341-2d4e7e60dbdf\") " pod="openshift-must-gather-5vzlx/perf-node-gather-daemonset-rz8tn" Sep 09 23:18:36.256301 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:36.256273 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bflw\" (UniqueName: \"kubernetes.io/projected/715f2358-eb71-4e18-9341-2d4e7e60dbdf-kube-api-access-6bflw\") pod \"perf-node-gather-daemonset-rz8tn\" (UID: \"715f2358-eb71-4e18-9341-2d4e7e60dbdf\") " pod="openshift-must-gather-5vzlx/perf-node-gather-daemonset-rz8tn" Sep 09 23:18:36.256448 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:36.256299 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/715f2358-eb71-4e18-9341-2d4e7e60dbdf-lib-modules\") pod \"perf-node-gather-daemonset-rz8tn\" (UID: \"715f2358-eb71-4e18-9341-2d4e7e60dbdf\") " pod="openshift-must-gather-5vzlx/perf-node-gather-daemonset-rz8tn" Sep 09 23:18:36.357048 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:36.357012 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/715f2358-eb71-4e18-9341-2d4e7e60dbdf-sys\") pod \"perf-node-gather-daemonset-rz8tn\" (UID: \"715f2358-eb71-4e18-9341-2d4e7e60dbdf\") " pod="openshift-must-gather-5vzlx/perf-node-gather-daemonset-rz8tn" Sep 09 23:18:36.357048 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:36.357046 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bflw\" (UniqueName: \"kubernetes.io/projected/715f2358-eb71-4e18-9341-2d4e7e60dbdf-kube-api-access-6bflw\") pod \"perf-node-gather-daemonset-rz8tn\" (UID: \"715f2358-eb71-4e18-9341-2d4e7e60dbdf\") " pod="openshift-must-gather-5vzlx/perf-node-gather-daemonset-rz8tn" Sep 09 23:18:36.357302 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:36.357065 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/715f2358-eb71-4e18-9341-2d4e7e60dbdf-lib-modules\") pod \"perf-node-gather-daemonset-rz8tn\" (UID: \"715f2358-eb71-4e18-9341-2d4e7e60dbdf\") " pod="openshift-must-gather-5vzlx/perf-node-gather-daemonset-rz8tn" Sep 09 23:18:36.357302 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:36.357153 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/715f2358-eb71-4e18-9341-2d4e7e60dbdf-sys\") pod \"perf-node-gather-daemonset-rz8tn\" (UID: \"715f2358-eb71-4e18-9341-2d4e7e60dbdf\") " pod="openshift-must-gather-5vzlx/perf-node-gather-daemonset-rz8tn" Sep 09 23:18:36.357302 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:36.357166 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/715f2358-eb71-4e18-9341-2d4e7e60dbdf-proc\") pod \"perf-node-gather-daemonset-rz8tn\" (UID: \"715f2358-eb71-4e18-9341-2d4e7e60dbdf\") " pod="openshift-must-gather-5vzlx/perf-node-gather-daemonset-rz8tn" Sep 09 23:18:36.357302 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:36.357230 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/715f2358-eb71-4e18-9341-2d4e7e60dbdf-proc\") pod \"perf-node-gather-daemonset-rz8tn\" (UID: \"715f2358-eb71-4e18-9341-2d4e7e60dbdf\") " pod="openshift-must-gather-5vzlx/perf-node-gather-daemonset-rz8tn" Sep 09 23:18:36.357302 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:36.357235 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/715f2358-eb71-4e18-9341-2d4e7e60dbdf-podres\") pod \"perf-node-gather-daemonset-rz8tn\" (UID: \"715f2358-eb71-4e18-9341-2d4e7e60dbdf\") " pod="openshift-must-gather-5vzlx/perf-node-gather-daemonset-rz8tn" Sep 09 23:18:36.357302 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:36.357299 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/715f2358-eb71-4e18-9341-2d4e7e60dbdf-lib-modules\") pod \"perf-node-gather-daemonset-rz8tn\" (UID: \"715f2358-eb71-4e18-9341-2d4e7e60dbdf\") " pod="openshift-must-gather-5vzlx/perf-node-gather-daemonset-rz8tn" Sep 09 23:18:36.357564 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:36.357326 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/715f2358-eb71-4e18-9341-2d4e7e60dbdf-podres\") pod \"perf-node-gather-daemonset-rz8tn\" (UID: \"715f2358-eb71-4e18-9341-2d4e7e60dbdf\") " pod="openshift-must-gather-5vzlx/perf-node-gather-daemonset-rz8tn" Sep 09 23:18:36.365879 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:36.365854 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bflw\" (UniqueName: \"kubernetes.io/projected/715f2358-eb71-4e18-9341-2d4e7e60dbdf-kube-api-access-6bflw\") pod \"perf-node-gather-daemonset-rz8tn\" (UID: \"715f2358-eb71-4e18-9341-2d4e7e60dbdf\") " pod="openshift-must-gather-5vzlx/perf-node-gather-daemonset-rz8tn" Sep 09 23:18:36.480280 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:36.480205 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5vzlx/perf-node-gather-daemonset-rz8tn" Sep 09 23:18:36.750290 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:36.750260 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5vzlx/perf-node-gather-daemonset-rz8tn"] Sep 09 23:18:36.751800 ip-10-0-139-252 kubenswrapper[2578]: W0909 23:18:36.751769 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod715f2358_eb71_4e18_9341_2d4e7e60dbdf.slice/crio-8f5a92c4ddfd564b73b01f3be8f519943eefaa835d584e0c1f6fc2db0d337397 WatchSource:0}: Error finding container 8f5a92c4ddfd564b73b01f3be8f519943eefaa835d584e0c1f6fc2db0d337397: Status 404 returned error can't find the container with id 8f5a92c4ddfd564b73b01f3be8f519943eefaa835d584e0c1f6fc2db0d337397 Sep 09 23:18:36.928249 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:36.928222 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-jfnt9_b0439d7f-12cb-477c-965e-39e908f129ba/volume-data-source-validator/0.log" Sep 09 23:18:37.032997 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:37.032955 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5vzlx/perf-node-gather-daemonset-rz8tn" event={"ID":"715f2358-eb71-4e18-9341-2d4e7e60dbdf","Type":"ContainerStarted","Data":"22bea6408450d4850f4f4b6c9965ec4ed21b75c3cf70b91064216821ed24599d"} Sep 09 23:18:37.033176 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:37.033014 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5vzlx/perf-node-gather-daemonset-rz8tn" event={"ID":"715f2358-eb71-4e18-9341-2d4e7e60dbdf","Type":"ContainerStarted","Data":"8f5a92c4ddfd564b73b01f3be8f519943eefaa835d584e0c1f6fc2db0d337397"} Sep 09 23:18:37.033243 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:37.033179 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5vzlx/perf-node-gather-daemonset-rz8tn" Sep 09 23:18:37.050999 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:37.050952 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5vzlx/perf-node-gather-daemonset-rz8tn" podStartSLOduration=1.050938956 podStartE2EDuration="1.050938956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:18:37.048791433 +0000 UTC m=+1177.799376361" watchObservedRunningTime="2026-09-09 23:18:37.050938956 +0000 UTC m=+1177.801523882" Sep 09 23:18:38.180208 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:38.180171 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wwkz5_e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec/dns/0.log" Sep 09 23:18:38.201470 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:38.201446 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wwkz5_e0c19faf-3ecc-4c86-8bb6-f12f2de6cbec/kube-rbac-proxy/0.log" Sep 09 23:18:38.224015 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:38.223991 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6j767_6c9f70f3-174f-4bcf-9bb3-838910f4fce9/dns-node-resolver/0.log" Sep 09 23:18:38.812695 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:38.812660 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-596d7d6d66-sl4ws_d9701aef-6851-4a0d-9381-a7625bbd0cc5/registry/0.log" Sep 09 23:18:38.832216 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:38.832193 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-75rcr_c864525a-b22a-4fef-ba7e-4b9541cbd717/node-ca/0.log" Sep 09 23:18:39.885349 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:39.885324 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7db58bdd4-lkdr8_5afd01d0-a070-4844-8eff-a6447e22b7d5/kube-auth-proxy/0.log" Sep 09 23:18:39.916358 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:39.916332 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lzj2s_1a051327-0bb5-4299-a04e-b402de10abf8/istio-proxy/1.log" Sep 09 23:18:39.917121 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:39.917097 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lzj2s_1a051327-0bb5-4299-a04e-b402de10abf8/istio-proxy/0.log" Sep 09 23:18:39.943064 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:39.943039 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-79k7l_70f6be58-73a2-4599-87f6-b125b9a46d0d/payload-pre-processing/0.log" Sep 09 23:18:39.968015 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:39.967990 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-wd9cw_43afd992-8822-4f8b-a80e-02a5ffd0fc0a/payload-processing/0.log" Sep 09 23:18:40.629151 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:40.629121 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9prb4_7d14c654-5bd8-4b10-aef1-32bf396708ab/serve-healthcheck-canary/0.log" Sep 09 23:18:41.139483 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:41.139453 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-7q6zd_35235a21-db5c-4908-971d-af808f0e4441/insights-operator/1.log" Sep 09 23:18:41.139914 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:41.139651 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-7q6zd_35235a21-db5c-4908-971d-af808f0e4441/insights-operator/0.log" Sep 09 23:18:41.329854 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:41.329824 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vltnr_92396342-28ed-43ed-b24d-607e3855829d/kube-rbac-proxy/0.log" Sep 09 23:18:41.352007 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:41.351980 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vltnr_92396342-28ed-43ed-b24d-607e3855829d/exporter/0.log" Sep 09 23:18:41.374774 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:41.374743 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vltnr_92396342-28ed-43ed-b24d-607e3855829d/extractor/0.log" Sep 09 23:18:43.049528 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:43.049499 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5vzlx/perf-node-gather-daemonset-rz8tn" Sep 09 23:18:43.465916 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:43.465818 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-4sss6_329724e3-d1fb-46c9-b267-061c965cede0/manager/0.log" Sep 09 23:18:43.509505 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:43.509477 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-wzvqr_3828e5a3-af2d-4422-8fea-b9fd6fafd1ef/manager/0.log" Sep 09 23:18:43.561753 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:43.561721 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-rb4ng_94b353b9-a213-40d3-993f-ac949389e2f4/server/0.log" Sep 09 23:18:43.672274 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:43.672244 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65d8796fc4-9t9t4_df244f8e-1b94-47ef-9c41-f0ce67302a5d/manager/0.log" Sep 09 23:18:43.690698 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:43.690678 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65d8796fc4-9t9t4_df244f8e-1b94-47ef-9c41-f0ce67302a5d/copy-manifests/0.log" Sep 09 23:18:43.846777 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:43.846750 2578 scope.go:117] "RemoveContainer" containerID="d155d233f991938dc03aa4045d75dd8d5860de479c5960c2edb694846ae42a91" Sep 09 23:18:43.846950 ip-10-0-139-252 kubenswrapper[2578]: E0909 23:18:43.846933 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg_llm(5cbd14d2-0fc0-4d69-a7ca-001692735fd9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-4psfg" podUID="5cbd14d2-0fc0-4d69-a7ca-001692735fd9" Sep 09 23:18:46.583926 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:46.583899 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7db58bdd4-lkdr8_5afd01d0-a070-4844-8eff-a6447e22b7d5/kube-auth-proxy/0.log" Sep 09 23:18:46.602035 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:46.602003 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lzj2s_1a051327-0bb5-4299-a04e-b402de10abf8/istio-proxy/1.log" Sep 09 23:18:46.602646 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:46.602625 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lzj2s_1a051327-0bb5-4299-a04e-b402de10abf8/istio-proxy/0.log" Sep 09 23:18:46.614143 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:46.614076 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-79k7l_70f6be58-73a2-4599-87f6-b125b9a46d0d/payload-pre-processing/0.log" Sep 09 23:18:46.630520 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:46.630499 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-wd9cw_43afd992-8822-4f8b-a80e-02a5ffd0fc0a/payload-processing/0.log" Sep 09 23:18:47.897433 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:47.897306 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7db58bdd4-lkdr8_5afd01d0-a070-4844-8eff-a6447e22b7d5/kube-auth-proxy/0.log" Sep 09 23:18:47.921714 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:47.921677 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lzj2s_1a051327-0bb5-4299-a04e-b402de10abf8/istio-proxy/1.log" Sep 09 23:18:47.922324 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:47.922302 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lzj2s_1a051327-0bb5-4299-a04e-b402de10abf8/istio-proxy/0.log" Sep 09 23:18:47.934357 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:47.934338 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-79k7l_70f6be58-73a2-4599-87f6-b125b9a46d0d/payload-pre-processing/0.log" Sep 09 23:18:47.950602 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:47.950578 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-wd9cw_43afd992-8822-4f8b-a80e-02a5ffd0fc0a/payload-processing/0.log" Sep 09 23:18:50.317929 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:50.317895 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-xrqwx_9a87380f-0c72-4392-9f94-95c1f3ba977f/migrator/0.log" Sep 09 23:18:50.342283 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:50.342260 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-xrqwx_9a87380f-0c72-4392-9f94-95c1f3ba977f/graceful-termination/0.log" Sep 09 23:18:51.861433 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:51.861407 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kpcwm_ec88f986-9c08-4e93-89c5-919467243b6f/kube-multus-additional-cni-plugins/0.log" Sep 09 23:18:51.886804 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:51.886771 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kpcwm_ec88f986-9c08-4e93-89c5-919467243b6f/egress-router-binary-copy/0.log" Sep 09 23:18:51.904750 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:51.904727 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kpcwm_ec88f986-9c08-4e93-89c5-919467243b6f/cni-plugins/0.log" Sep 09 23:18:51.928728 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:51.928703 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kpcwm_ec88f986-9c08-4e93-89c5-919467243b6f/bond-cni-plugin/0.log" Sep 09 23:18:51.951307 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:51.951285 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kpcwm_ec88f986-9c08-4e93-89c5-919467243b6f/routeoverride-cni/0.log" Sep 09 23:18:51.974936 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:51.974916 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kpcwm_ec88f986-9c08-4e93-89c5-919467243b6f/whereabouts-cni-bincopy/0.log" Sep 09 23:18:51.996815 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:51.996794 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kpcwm_ec88f986-9c08-4e93-89c5-919467243b6f/whereabouts-cni/0.log" Sep 09 23:18:52.223669 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:52.223585 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dsd8k_b33b6c26-be92-4d99-8cbc-cc2ca4b7f65d/kube-multus/0.log" Sep 09 23:18:52.405694 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:52.405651 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-l9h9s_185f1e13-d2f1-4c57-99d8-c3eb63e20512/network-metrics-daemon/0.log" Sep 09 23:18:52.426649 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:52.426621 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-l9h9s_185f1e13-d2f1-4c57-99d8-c3eb63e20512/kube-rbac-proxy/0.log" Sep 09 23:18:53.291668 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:53.291641 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vtjn_c3cbf093-0861-4893-8d8c-8ca3d30987c2/ovn-controller/0.log" Sep 09 23:18:53.311522 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:53.311501 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vtjn_c3cbf093-0861-4893-8d8c-8ca3d30987c2/ovn-acl-logging/0.log" Sep 09 23:18:53.344124 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:53.344089 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vtjn_c3cbf093-0861-4893-8d8c-8ca3d30987c2/kube-rbac-proxy-node/0.log" Sep 09 23:18:53.370532 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:53.370508 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vtjn_c3cbf093-0861-4893-8d8c-8ca3d30987c2/kube-rbac-proxy-ovn-metrics/0.log" Sep 09 23:18:53.387763 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:53.387739 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vtjn_c3cbf093-0861-4893-8d8c-8ca3d30987c2/northd/0.log" Sep 09 23:18:53.410837 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:53.410815 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vtjn_c3cbf093-0861-4893-8d8c-8ca3d30987c2/nbdb/0.log" Sep 09 23:18:53.432176 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:53.432155 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vtjn_c3cbf093-0861-4893-8d8c-8ca3d30987c2/sbdb/0.log" Sep 09 23:18:53.488902 ip-10-0-139-252 kubenswrapper[2578]: I0909 23:18:53.488869 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4vtjn_c3cbf093-0861-4893-8d8c-8ca3d30987c2/ovnkube-controller/0.log"