Sep 09 20:50:16.621521 ip-10-0-136-126 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 09 20:50:16.621533 ip-10-0-136-126 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 09 20:50:16.621541 ip-10-0-136-126 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 09 20:50:16.621695 ip-10-0-136-126 systemd[1]: Failed to start Kubernetes Kubelet. Sep 09 20:50:26.716082 ip-10-0-136-126 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 09 20:50:26.716101 ip-10-0-136-126 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot b5163b01c461442dbe44b67d9542cad0 -- Sep 09 20:52:18.836279 ip-10-0-136-126 systemd[1]: Starting Kubernetes Kubelet... Sep 09 20:52:19.252035 ip-10-0-136-126 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 20:52:19.252035 ip-10-0-136-126 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 09 20:52:19.252035 ip-10-0-136-126 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 20:52:19.252035 ip-10-0-136-126 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 09 20:52:19.252035 ip-10-0-136-126 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 20:52:19.253143 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.253041 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 09 20:52:19.258456 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258431 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 20:52:19.258456 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258451 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 20:52:19.258456 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258455 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 20:52:19.258456 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258460 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 20:52:19.258855 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258469 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 20:52:19.258855 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258473 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 20:52:19.258855 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258479 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 20:52:19.258855 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258485 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 20:52:19.258855 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258490 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 20:52:19.258855 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258493 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 20:52:19.258855 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258497 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 20:52:19.258855 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258500 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 20:52:19.258855 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258504 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 20:52:19.258855 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258508 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 20:52:19.258855 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258512 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 20:52:19.258855 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258516 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 20:52:19.258855 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258520 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 20:52:19.258855 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258523 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 20:52:19.258855 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258527 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 20:52:19.258855 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258532 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 20:52:19.258855 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258537 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 20:52:19.258855 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258540 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 20:52:19.258855 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258545 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 20:52:19.258855 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258548 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 20:52:19.259687 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258552 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 20:52:19.259687 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258556 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 20:52:19.259687 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258559 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 20:52:19.259687 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258563 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 09 20:52:19.259687 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258567 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 20:52:19.259687 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258571 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 20:52:19.259687 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258575 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 20:52:19.259687 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258579 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 20:52:19.259687 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258583 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 20:52:19.259687 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258589 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 20:52:19.259687 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258594 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 20:52:19.259687 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258598 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 20:52:19.259687 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258602 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 20:52:19.259687 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258607 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 20:52:19.259687 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258612 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 20:52:19.259687 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258616 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 20:52:19.259687 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258620 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 20:52:19.259687 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258624 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 20:52:19.259687 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258628 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 20:52:19.260284 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258632 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 20:52:19.260284 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258637 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 20:52:19.260284 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258641 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 20:52:19.260284 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258645 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 20:52:19.260284 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258649 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 20:52:19.260284 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258653 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 20:52:19.260284 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258657 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 20:52:19.260284 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258661 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 20:52:19.260284 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258665 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 20:52:19.260284 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258669 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 20:52:19.260284 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258673 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 20:52:19.260284 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258677 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 20:52:19.260284 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258680 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 20:52:19.260284 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258685 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 20:52:19.260284 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258689 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 20:52:19.260284 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258693 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 20:52:19.260284 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258697 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 20:52:19.260284 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258701 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 20:52:19.260284 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258704 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 20:52:19.260284 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258710 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 20:52:19.260813 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258713 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 20:52:19.260813 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258719 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 20:52:19.260813 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258723 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 20:52:19.260813 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258727 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 20:52:19.260813 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258730 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 20:52:19.260813 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258735 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 20:52:19.260813 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258741 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 20:52:19.260813 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258746 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 20:52:19.260813 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258750 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 20:52:19.260813 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258757 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 20:52:19.260813 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258763 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 20:52:19.260813 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258768 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 20:52:19.260813 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258773 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 20:52:19.260813 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258777 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 20:52:19.260813 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258782 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 20:52:19.260813 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258786 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 20:52:19.260813 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258790 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 20:52:19.260813 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258795 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 20:52:19.260813 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258802 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 20:52:19.261298 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258807 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 20:52:19.261298 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258812 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 20:52:19.261298 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258816 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 20:52:19.261298 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258821 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 20:52:19.261298 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258825 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 20:52:19.261298 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258829 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 20:52:19.261298 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258834 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 20:52:19.261298 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258839 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 20:52:19.261298 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258843 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 20:52:19.261298 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258848 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 20:52:19.261298 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258852 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 20:52:19.261298 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258856 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 20:52:19.261298 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258860 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 20:52:19.261298 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258864 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 20:52:19.261298 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258869 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 20:52:19.261298 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258874 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 20:52:19.261298 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.258878 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 20:52:19.261298 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260107 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 20:52:19.261298 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260119 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 20:52:19.261298 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260122 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 20:52:19.261790 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260125 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 20:52:19.261790 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260128 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 20:52:19.261790 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260130 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 20:52:19.261790 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260133 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 20:52:19.261790 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260136 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 20:52:19.261790 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260139 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 20:52:19.261790 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260143 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 20:52:19.261790 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260146 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 20:52:19.261790 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260151 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 20:52:19.261790 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260155 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 20:52:19.261790 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260158 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 20:52:19.261790 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260161 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 20:52:19.261790 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260165 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 09 20:52:19.261790 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260167 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 20:52:19.261790 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260171 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 20:52:19.261790 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260174 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 20:52:19.261790 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260177 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 20:52:19.261790 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260180 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 20:52:19.261790 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260182 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 20:52:19.262286 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260186 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 20:52:19.262286 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260190 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 20:52:19.262286 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260193 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 20:52:19.262286 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260196 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 20:52:19.262286 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260199 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 20:52:19.262286 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260202 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 20:52:19.262286 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260205 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 20:52:19.262286 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260208 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 20:52:19.262286 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260210 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 20:52:19.262286 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260214 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 20:52:19.262286 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260217 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 20:52:19.262286 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260220 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 20:52:19.262286 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260223 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 20:52:19.262286 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260226 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 20:52:19.262286 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260229 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 20:52:19.262286 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260232 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 20:52:19.262286 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260234 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 20:52:19.262286 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260237 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 20:52:19.262286 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260239 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 20:52:19.262757 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260242 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 20:52:19.262757 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260245 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 20:52:19.262757 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260247 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 20:52:19.262757 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260250 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 20:52:19.262757 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260253 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 20:52:19.262757 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260256 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 20:52:19.262757 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260258 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 20:52:19.262757 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260261 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 20:52:19.262757 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260265 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 20:52:19.262757 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260269 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 20:52:19.262757 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260272 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 20:52:19.262757 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260274 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 20:52:19.262757 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260277 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 20:52:19.262757 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260279 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 20:52:19.262757 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260282 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 20:52:19.262757 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260285 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 20:52:19.262757 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260288 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 20:52:19.262757 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260290 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 20:52:19.262757 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260293 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 20:52:19.262757 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260296 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 20:52:19.263277 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260299 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 20:52:19.263277 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260301 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 20:52:19.263277 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260304 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 20:52:19.263277 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260307 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 20:52:19.263277 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260310 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 20:52:19.263277 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260313 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 20:52:19.263277 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260316 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 20:52:19.263277 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260319 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 20:52:19.263277 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260321 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 20:52:19.263277 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260324 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 20:52:19.263277 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260327 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 20:52:19.263277 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260329 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 20:52:19.263277 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260332 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 20:52:19.263277 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260334 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 20:52:19.263277 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260337 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 20:52:19.263277 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260340 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 20:52:19.263277 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260342 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 20:52:19.263277 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260345 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 20:52:19.263277 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260349 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 20:52:19.263745 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260351 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 20:52:19.263745 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260354 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 20:52:19.263745 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260357 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 20:52:19.263745 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260359 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 20:52:19.263745 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260362 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 20:52:19.263745 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260364 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 20:52:19.263745 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260367 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 20:52:19.263745 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260369 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 20:52:19.263745 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260372 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 20:52:19.263745 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260375 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 20:52:19.263745 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260377 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 20:52:19.263745 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260379 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 20:52:19.263745 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260382 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 20:52:19.263745 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260384 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 20:52:19.263745 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260387 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 20:52:19.263745 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260390 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 20:52:19.263745 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260392 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 20:52:19.263745 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260403 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 20:52:19.263745 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260407 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 20:52:19.263745 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260915 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 20:52:19.264246 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260921 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 20:52:19.264246 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260924 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 20:52:19.264246 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260942 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 20:52:19.264246 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260946 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 20:52:19.264246 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260949 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 20:52:19.264246 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260952 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 20:52:19.264246 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260954 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 20:52:19.264246 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260957 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 20:52:19.264246 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260960 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 20:52:19.264246 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260963 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 20:52:19.264246 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260965 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 20:52:19.264246 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260968 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 20:52:19.264246 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260970 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 20:52:19.264246 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260973 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 20:52:19.264246 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260976 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 20:52:19.264246 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260978 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 20:52:19.264246 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260981 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 20:52:19.264246 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260983 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 20:52:19.264246 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260986 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 20:52:19.264246 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260988 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 20:52:19.264730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260991 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 20:52:19.264730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260994 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 20:52:19.264730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260996 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 20:52:19.264730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.260999 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 20:52:19.264730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261002 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 20:52:19.264730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261004 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 20:52:19.264730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261007 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 20:52:19.264730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261009 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 20:52:19.264730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261012 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 20:52:19.264730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261021 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 20:52:19.264730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261026 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 20:52:19.264730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261030 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 20:52:19.264730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261033 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 20:52:19.264730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261036 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 20:52:19.264730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261039 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 20:52:19.264730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261041 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 20:52:19.264730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261044 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 20:52:19.264730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261047 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 20:52:19.264730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261049 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 20:52:19.264730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261052 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 20:52:19.265225 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261056 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 20:52:19.265225 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261060 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 20:52:19.265225 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261063 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 20:52:19.265225 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261066 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 20:52:19.265225 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261069 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 20:52:19.265225 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261072 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 20:52:19.265225 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261075 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 20:52:19.265225 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261078 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 20:52:19.265225 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261081 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 20:52:19.265225 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261084 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 20:52:19.265225 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261086 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 20:52:19.265225 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261089 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 20:52:19.265225 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261091 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 20:52:19.265225 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261094 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 20:52:19.265225 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261097 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 20:52:19.265225 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261100 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 20:52:19.265225 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261102 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 09 20:52:19.265225 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261105 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 20:52:19.265225 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261107 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 20:52:19.265730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261110 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 20:52:19.265730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261115 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 20:52:19.265730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261118 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 20:52:19.265730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261126 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 20:52:19.265730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261129 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 20:52:19.265730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261132 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 20:52:19.265730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261135 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 20:52:19.265730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261137 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 20:52:19.265730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261139 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 20:52:19.265730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261143 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 20:52:19.265730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261145 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 20:52:19.265730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261148 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 20:52:19.265730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261151 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 20:52:19.265730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261153 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 20:52:19.265730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261156 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 20:52:19.265730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261159 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 20:52:19.265730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261162 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 20:52:19.265730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261165 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 20:52:19.265730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261167 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 20:52:19.265730 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261170 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 20:52:19.266243 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261173 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 20:52:19.266243 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261175 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 20:52:19.266243 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261178 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 20:52:19.266243 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261181 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 20:52:19.266243 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261183 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 20:52:19.266243 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261186 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 20:52:19.266243 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261188 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 20:52:19.266243 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261191 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 20:52:19.266243 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261194 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 20:52:19.266243 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261196 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 20:52:19.266243 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261199 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 20:52:19.266243 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261201 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 20:52:19.266243 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261204 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 20:52:19.266243 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261206 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 20:52:19.266243 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261209 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 20:52:19.266243 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261211 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 20:52:19.266243 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261219 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 20:52:19.266243 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261222 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 20:52:19.266243 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.261225 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 20:52:19.266243 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261305 2562 flags.go:64] FLAG: --address="0.0.0.0" Sep 09 20:52:19.266729 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261344 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 09 20:52:19.266729 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261354 2562 flags.go:64] FLAG: --anonymous-auth="true" Sep 09 20:52:19.266729 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261359 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 09 20:52:19.266729 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261363 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 09 20:52:19.266729 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261367 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 09 20:52:19.266729 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261371 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 09 20:52:19.266729 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261375 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 09 20:52:19.266729 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261378 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 09 20:52:19.266729 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261381 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 09 20:52:19.266729 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261384 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 09 20:52:19.266729 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261388 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 09 20:52:19.266729 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261391 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 09 20:52:19.266729 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261395 2562 flags.go:64] FLAG: --cgroup-root="" Sep 09 20:52:19.266729 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261398 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 09 20:52:19.266729 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261401 2562 flags.go:64] FLAG: --client-ca-file="" Sep 09 20:52:19.266729 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261403 2562 flags.go:64] FLAG: --cloud-provider="external" Sep 09 20:52:19.266729 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261406 2562 flags.go:64] FLAG: --cluster-dns="[]" Sep 09 20:52:19.266729 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261413 2562 flags.go:64] FLAG: --cluster-domain="" Sep 09 20:52:19.266729 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261416 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 09 20:52:19.266729 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261419 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 09 20:52:19.266729 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261422 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 09 20:52:19.266729 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261426 2562 flags.go:64] FLAG: --container-log-max-files="5" Sep 09 20:52:19.266729 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261429 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 09 20:52:19.266729 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261433 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 09 20:52:19.267325 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261436 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 09 20:52:19.267325 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261439 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 09 20:52:19.267325 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261442 2562 flags.go:64] FLAG: --contention-profiling="false" Sep 09 20:52:19.267325 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261445 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 09 20:52:19.267325 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261448 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 09 20:52:19.267325 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261457 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 09 20:52:19.267325 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261460 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 09 20:52:19.267325 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261464 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 09 20:52:19.267325 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261467 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 09 20:52:19.267325 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261470 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 09 20:52:19.267325 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261480 2562 flags.go:64] FLAG: --enable-load-reader="false" Sep 09 20:52:19.267325 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261485 2562 flags.go:64] FLAG: --enable-server="true" Sep 09 20:52:19.267325 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261488 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 09 20:52:19.267325 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261496 2562 flags.go:64] FLAG: --event-burst="100" Sep 09 20:52:19.267325 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261499 2562 flags.go:64] FLAG: --event-qps="50" Sep 09 20:52:19.267325 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261502 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 09 20:52:19.267325 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261505 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 09 20:52:19.267325 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261508 2562 flags.go:64] FLAG: --eviction-hard="" Sep 09 20:52:19.267325 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261515 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 09 20:52:19.267325 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261518 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 09 20:52:19.267325 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261521 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 09 20:52:19.267325 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261524 2562 flags.go:64] FLAG: --eviction-soft="" Sep 09 20:52:19.267325 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261527 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 09 20:52:19.267325 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261530 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 09 20:52:19.267325 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261533 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 09 20:52:19.267957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261536 2562 flags.go:64] FLAG: --experimental-mounter-path="" Sep 09 20:52:19.267957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261539 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 09 20:52:19.267957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261541 2562 flags.go:64] FLAG: --fail-swap-on="true" Sep 09 20:52:19.267957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261544 2562 flags.go:64] FLAG: --feature-gates="" Sep 09 20:52:19.267957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261548 2562 flags.go:64] FLAG: --file-check-frequency="20s" Sep 09 20:52:19.267957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261552 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 09 20:52:19.267957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261555 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 09 20:52:19.267957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261559 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 09 20:52:19.267957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261562 2562 flags.go:64] FLAG: --healthz-port="10248" Sep 09 20:52:19.267957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261565 2562 flags.go:64] FLAG: --help="false" Sep 09 20:52:19.267957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261568 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-136-126.ec2.internal" Sep 09 20:52:19.267957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261571 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 09 20:52:19.267957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261574 2562 flags.go:64] FLAG: --http-check-frequency="20s" Sep 09 20:52:19.267957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261583 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 09 20:52:19.267957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261587 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 09 20:52:19.267957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261592 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 09 20:52:19.267957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261595 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 09 20:52:19.267957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261598 2562 flags.go:64] FLAG: --image-service-endpoint="" Sep 09 20:52:19.267957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261600 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 09 20:52:19.267957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261603 2562 flags.go:64] FLAG: --kube-api-burst="100" Sep 09 20:52:19.267957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261606 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 09 20:52:19.267957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261609 2562 flags.go:64] FLAG: --kube-api-qps="50" Sep 09 20:52:19.267957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261612 2562 flags.go:64] FLAG: --kube-reserved="" Sep 09 20:52:19.267957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261615 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 09 20:52:19.268582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261618 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 09 20:52:19.268582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261623 2562 flags.go:64] FLAG: --kubelet-cgroups="" Sep 09 20:52:19.268582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261626 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 09 20:52:19.268582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261629 2562 flags.go:64] FLAG: --lock-file="" Sep 09 20:52:19.268582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261631 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 09 20:52:19.268582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261634 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 09 20:52:19.268582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261638 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 09 20:52:19.268582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261643 2562 flags.go:64] FLAG: --log-json-split-stream="false" Sep 09 20:52:19.268582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261646 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 09 20:52:19.268582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261649 2562 flags.go:64] FLAG: --log-text-split-stream="false" Sep 09 20:52:19.268582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261652 2562 flags.go:64] FLAG: --logging-format="text" Sep 09 20:52:19.268582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261655 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 09 20:52:19.268582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261659 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 09 20:52:19.268582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261662 2562 flags.go:64] FLAG: --manifest-url="" Sep 09 20:52:19.268582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261665 2562 flags.go:64] FLAG: --manifest-url-header="" Sep 09 20:52:19.268582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261669 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 09 20:52:19.268582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261675 2562 flags.go:64] FLAG: --max-open-files="1000000" Sep 09 20:52:19.268582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261679 2562 flags.go:64] FLAG: --max-pods="110" Sep 09 20:52:19.268582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261682 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 09 20:52:19.268582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261685 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 09 20:52:19.268582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261688 2562 flags.go:64] FLAG: --memory-manager-policy="None" Sep 09 20:52:19.268582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261691 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 09 20:52:19.268582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261700 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 09 20:52:19.268582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261705 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 09 20:52:19.268582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261708 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 09 20:52:19.269258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261715 2562 flags.go:64] FLAG: --node-status-max-images="50" Sep 09 20:52:19.269258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261718 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 09 20:52:19.269258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261721 2562 flags.go:64] FLAG: --oom-score-adj="-999" Sep 09 20:52:19.269258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261724 2562 flags.go:64] FLAG: --pod-cidr="" Sep 09 20:52:19.269258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261727 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 09 20:52:19.269258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261734 2562 flags.go:64] FLAG: --pod-manifest-path="" Sep 09 20:52:19.269258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261737 2562 flags.go:64] FLAG: --pod-max-pids="-1" Sep 09 20:52:19.269258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261740 2562 flags.go:64] FLAG: --pods-per-core="0" Sep 09 20:52:19.269258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261743 2562 flags.go:64] FLAG: --port="10250" Sep 09 20:52:19.269258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261748 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 09 20:52:19.269258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261751 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05b7de2726a149701" Sep 09 20:52:19.269258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261754 2562 flags.go:64] FLAG: --qos-reserved="" Sep 09 20:52:19.269258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261757 2562 flags.go:64] FLAG: --read-only-port="10255" Sep 09 20:52:19.269258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261760 2562 flags.go:64] FLAG: --register-node="true" Sep 09 20:52:19.269258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261763 2562 flags.go:64] FLAG: --register-with-taints="" Sep 09 20:52:19.269258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261767 2562 flags.go:64] FLAG: --registry-burst="10" Sep 09 20:52:19.269258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261770 2562 flags.go:64] FLAG: --registry-qps="5" Sep 09 20:52:19.269258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261773 2562 flags.go:64] FLAG: --reserved-cpus="" Sep 09 20:52:19.269258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261776 2562 flags.go:64] FLAG: --reserved-memory="" Sep 09 20:52:19.269258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261779 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 09 20:52:19.269258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261782 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 09 20:52:19.269258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261785 2562 flags.go:64] FLAG: --rotate-certificates="false" Sep 09 20:52:19.269258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261788 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 09 20:52:19.269258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261791 2562 flags.go:64] FLAG: --runonce="false" Sep 09 20:52:19.269258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261794 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 09 20:52:19.269888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261797 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 09 20:52:19.269888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261800 2562 flags.go:64] FLAG: --seccomp-default="false" Sep 09 20:52:19.269888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261803 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 09 20:52:19.269888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261806 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 09 20:52:19.269888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261809 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 09 20:52:19.269888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261821 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 09 20:52:19.269888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261824 2562 flags.go:64] FLAG: --storage-driver-password="root" Sep 09 20:52:19.269888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261827 2562 flags.go:64] FLAG: --storage-driver-secure="false" Sep 09 20:52:19.269888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261830 2562 flags.go:64] FLAG: --storage-driver-table="stats" Sep 09 20:52:19.269888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261833 2562 flags.go:64] FLAG: --storage-driver-user="root" Sep 09 20:52:19.269888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261835 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 09 20:52:19.269888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261839 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 09 20:52:19.269888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261841 2562 flags.go:64] FLAG: --system-cgroups="" Sep 09 20:52:19.269888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261844 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 09 20:52:19.269888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261850 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 09 20:52:19.269888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261853 2562 flags.go:64] FLAG: --tls-cert-file="" Sep 09 20:52:19.269888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261857 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 09 20:52:19.269888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261863 2562 flags.go:64] FLAG: --tls-min-version="" Sep 09 20:52:19.269888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261866 2562 flags.go:64] FLAG: --tls-private-key-file="" Sep 09 20:52:19.269888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261869 2562 flags.go:64] FLAG: --topology-manager-policy="none" Sep 09 20:52:19.269888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261872 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 09 20:52:19.269888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261875 2562 flags.go:64] FLAG: --topology-manager-scope="container" Sep 09 20:52:19.269888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261878 2562 flags.go:64] FLAG: --v="2" Sep 09 20:52:19.269888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261882 2562 flags.go:64] FLAG: --version="false" Sep 09 20:52:19.269888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261886 2562 flags.go:64] FLAG: --vmodule="" Sep 09 20:52:19.270768 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261890 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 09 20:52:19.270768 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.261893 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 09 20:52:19.270768 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262076 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 20:52:19.270768 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262082 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 20:52:19.270768 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262085 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 20:52:19.270768 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262088 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 20:52:19.270768 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262091 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 20:52:19.270768 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262094 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 20:52:19.270768 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262097 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 20:52:19.270768 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262099 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 20:52:19.270768 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262102 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 20:52:19.270768 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262105 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 20:52:19.270768 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262109 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 20:52:19.270768 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262118 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 20:52:19.270768 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262120 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 20:52:19.270768 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262123 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 20:52:19.270768 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262126 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 20:52:19.270768 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262129 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 20:52:19.270768 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262131 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 20:52:19.270768 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262134 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 20:52:19.271559 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262136 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 20:52:19.271559 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262139 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 20:52:19.271559 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262142 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 20:52:19.271559 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262146 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 20:52:19.271559 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262148 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 09 20:52:19.271559 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262156 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 20:52:19.271559 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262159 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 20:52:19.271559 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262162 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 20:52:19.271559 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262164 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 20:52:19.271559 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262167 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 20:52:19.271559 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262169 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 20:52:19.271559 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262172 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 20:52:19.271559 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262175 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 20:52:19.271559 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262177 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 20:52:19.271559 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262180 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 20:52:19.271559 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262182 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 20:52:19.271559 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262185 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 20:52:19.271559 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262187 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 20:52:19.271559 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262190 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 20:52:19.271559 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262193 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 20:52:19.272205 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262196 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 20:52:19.272205 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262198 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 20:52:19.272205 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262201 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 20:52:19.272205 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262204 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 20:52:19.272205 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262207 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 20:52:19.272205 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262210 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 20:52:19.272205 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262218 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 20:52:19.272205 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262221 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 20:52:19.272205 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262224 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 20:52:19.272205 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262228 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 20:52:19.272205 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262231 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 20:52:19.272205 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262234 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 20:52:19.272205 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262236 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 20:52:19.272205 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262239 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 20:52:19.272205 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262242 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 20:52:19.272205 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262246 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 20:52:19.272205 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262249 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 20:52:19.272205 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262253 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 20:52:19.272205 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262256 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 20:52:19.272667 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262260 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 20:52:19.272667 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262264 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 20:52:19.272667 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262267 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 20:52:19.272667 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262270 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 20:52:19.272667 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262273 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 20:52:19.272667 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262276 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 20:52:19.272667 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262279 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 20:52:19.272667 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262282 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 20:52:19.272667 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262284 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 20:52:19.272667 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262287 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 20:52:19.272667 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262290 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 20:52:19.272667 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262293 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 20:52:19.272667 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262296 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 20:52:19.272667 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262298 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 20:52:19.272667 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262301 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 20:52:19.272667 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262304 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 20:52:19.272667 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262306 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 20:52:19.272667 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262310 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 20:52:19.272667 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262313 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 20:52:19.273144 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262315 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 20:52:19.273144 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262324 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 20:52:19.273144 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262327 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 20:52:19.273144 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262329 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 20:52:19.273144 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262333 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 20:52:19.273144 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262337 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 20:52:19.273144 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262341 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 20:52:19.273144 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262345 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 20:52:19.273144 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262348 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 20:52:19.273144 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262353 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 20:52:19.273144 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262356 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 20:52:19.273144 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262358 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 20:52:19.273144 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262361 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 20:52:19.273144 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262364 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 20:52:19.273144 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262366 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 20:52:19.273144 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262369 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 20:52:19.273144 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262372 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 20:52:19.273144 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262374 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 20:52:19.273144 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262377 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 20:52:19.273144 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262380 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 20:52:19.273654 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262382 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 20:52:19.273654 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262385 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 20:52:19.273654 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.262387 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 20:52:19.273654 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.263148 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 20:52:19.273654 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.271875 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 09 20:52:19.273654 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.271893 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 09 20:52:19.273654 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.271913 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 09 20:52:19.273654 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.271922 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 09 20:52:19.273654 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272002 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 20:52:19.273654 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272008 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 20:52:19.273654 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272012 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 20:52:19.273654 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272015 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 20:52:19.273654 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272018 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 20:52:19.273654 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272020 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 20:52:19.274023 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272024 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 20:52:19.274023 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272027 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 20:52:19.274023 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272029 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 20:52:19.274023 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272033 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 20:52:19.274023 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272035 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 20:52:19.274023 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272038 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 20:52:19.274023 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272041 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 20:52:19.274023 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272044 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 20:52:19.274023 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272047 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 20:52:19.274023 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272050 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 20:52:19.274023 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272053 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 20:52:19.274023 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272057 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 20:52:19.274023 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272059 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 20:52:19.274023 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272062 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 20:52:19.274023 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272064 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 20:52:19.274023 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272067 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 20:52:19.274023 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272070 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 20:52:19.274023 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272073 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 20:52:19.274023 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272076 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 20:52:19.274023 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272079 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 20:52:19.274508 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272083 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 20:52:19.274508 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272087 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 20:52:19.274508 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272090 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 20:52:19.274508 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272094 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 20:52:19.274508 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272097 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 20:52:19.274508 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272100 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 20:52:19.274508 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272103 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 20:52:19.274508 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272106 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 20:52:19.274508 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272109 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 20:52:19.274508 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272113 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 20:52:19.274508 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272116 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 20:52:19.274508 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272119 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 20:52:19.274508 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272122 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 20:52:19.274508 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272124 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 20:52:19.274508 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272127 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 20:52:19.274508 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272130 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 20:52:19.274508 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272132 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 20:52:19.274508 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272135 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 20:52:19.274508 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272137 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 20:52:19.274974 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272139 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 20:52:19.274974 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272142 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 20:52:19.274974 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272145 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 20:52:19.274974 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272148 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 20:52:19.274974 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272151 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 20:52:19.274974 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272154 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 20:52:19.274974 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272156 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 20:52:19.274974 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272159 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 20:52:19.274974 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272162 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 20:52:19.274974 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272165 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 20:52:19.274974 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272167 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 20:52:19.274974 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272169 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 20:52:19.274974 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272172 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 20:52:19.274974 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272175 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 20:52:19.274974 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272177 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 20:52:19.274974 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272180 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 20:52:19.274974 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272182 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 20:52:19.274974 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272185 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 20:52:19.274974 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272188 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 20:52:19.274974 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272191 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 20:52:19.275472 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272193 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 20:52:19.275472 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272196 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 20:52:19.275472 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272199 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 20:52:19.275472 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272202 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 20:52:19.275472 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272204 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 20:52:19.275472 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272207 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 20:52:19.275472 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272209 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 20:52:19.275472 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272212 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 20:52:19.275472 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272215 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 20:52:19.275472 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272217 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 20:52:19.275472 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272220 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 09 20:52:19.275472 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272223 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 20:52:19.275472 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272226 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 20:52:19.275472 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272229 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 20:52:19.275472 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272232 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 20:52:19.275472 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272235 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 20:52:19.275472 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272238 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 20:52:19.275472 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272240 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 20:52:19.275472 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272243 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 20:52:19.275472 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272246 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 20:52:19.276069 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272248 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 20:52:19.276069 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272251 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 20:52:19.276069 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272254 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 20:52:19.276069 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272256 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 20:52:19.276069 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272259 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 20:52:19.276069 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272261 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 20:52:19.276069 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272264 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 20:52:19.276069 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272266 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 20:52:19.276069 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272270 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 20:52:19.276069 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272274 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 20:52:19.276069 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272277 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 20:52:19.276069 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272279 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 20:52:19.276069 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272282 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 20:52:19.276069 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272285 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 20:52:19.276069 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.272290 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 20:52:19.276444 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272408 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 20:52:19.276444 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272414 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 20:52:19.276444 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272417 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 20:52:19.276444 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272420 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 20:52:19.276444 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272423 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 20:52:19.276444 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272426 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 20:52:19.276444 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272429 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 20:52:19.276444 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272432 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 20:52:19.276444 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272435 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 20:52:19.276444 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272437 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 20:52:19.276444 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272440 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 20:52:19.276444 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272443 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 20:52:19.276444 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272446 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 20:52:19.276444 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272449 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 20:52:19.276444 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272452 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 20:52:19.276444 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272455 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 20:52:19.276444 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272458 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 20:52:19.276444 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272460 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 20:52:19.276444 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272463 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 20:52:19.276444 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272466 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 20:52:19.276938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272468 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 20:52:19.276938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272471 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 20:52:19.276938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272473 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 20:52:19.276938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272476 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 20:52:19.276938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272478 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 20:52:19.276938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272482 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 20:52:19.276938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272486 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 20:52:19.276938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272488 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 20:52:19.276938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272491 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 20:52:19.276938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272493 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 20:52:19.276938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272497 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 20:52:19.276938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272499 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 20:52:19.276938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272502 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 20:52:19.276938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272505 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 20:52:19.276938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272507 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 20:52:19.276938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272510 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 20:52:19.276938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272512 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 20:52:19.276938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272515 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 20:52:19.276938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272517 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 20:52:19.276938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272520 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 20:52:19.277460 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272522 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 20:52:19.277460 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272525 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 20:52:19.277460 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272527 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 20:52:19.277460 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272530 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 20:52:19.277460 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272533 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 20:52:19.277460 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272536 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 20:52:19.277460 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272540 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 20:52:19.277460 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272544 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 20:52:19.277460 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272546 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 20:52:19.277460 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272549 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 20:52:19.277460 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272551 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 20:52:19.277460 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272554 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 20:52:19.277460 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272556 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 09 20:52:19.277460 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272559 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 20:52:19.277460 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272562 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 20:52:19.277460 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272564 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 20:52:19.277460 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272567 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 20:52:19.277460 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272569 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 20:52:19.277460 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272572 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 20:52:19.277938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272574 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 20:52:19.277938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272577 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 20:52:19.277938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272579 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 20:52:19.277938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272582 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 20:52:19.277938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272584 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 20:52:19.277938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272587 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 20:52:19.277938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272591 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 20:52:19.277938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272594 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 20:52:19.277938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272598 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 20:52:19.277938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272600 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 20:52:19.277938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272603 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 20:52:19.277938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272606 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 20:52:19.277938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272609 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 20:52:19.277938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272612 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 20:52:19.277938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272614 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 20:52:19.277938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272616 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 20:52:19.277938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272619 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 20:52:19.277938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272622 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 20:52:19.277938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272625 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 20:52:19.277938 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272628 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 20:52:19.278432 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272630 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 20:52:19.278432 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272633 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 20:52:19.278432 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272635 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 20:52:19.278432 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272638 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 20:52:19.278432 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272641 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 20:52:19.278432 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272643 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 20:52:19.278432 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272646 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 20:52:19.278432 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272648 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 20:52:19.278432 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272651 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 20:52:19.278432 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272653 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 20:52:19.278432 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272656 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 20:52:19.278432 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272659 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 20:52:19.278432 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272661 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 20:52:19.278432 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272663 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 20:52:19.278432 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272666 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 20:52:19.278432 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272669 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 20:52:19.278432 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272672 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 20:52:19.278432 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272674 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 20:52:19.278432 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272677 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 20:52:19.278432 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:19.272680 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 20:52:19.278971 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.272685 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 20:52:19.278971 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.272818 2562 server.go:962] "Client rotation is on, will bootstrap in background" Sep 09 20:52:19.278971 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.278797 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 09 20:52:19.279832 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.279818 2562 server.go:1019] "Starting client certificate rotation" Sep 09 20:52:19.280019 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.279940 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 20:52:19.280019 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.279984 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 20:52:19.282413 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.282394 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 20:52:19.287116 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.287075 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 20:52:19.298088 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.298065 2562 log.go:25] "Validated CRI v1 runtime API" Sep 09 20:52:19.301071 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.301055 2562 log.go:25] "Validated CRI v1 image API" Sep 09 20:52:19.302517 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.302501 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 09 20:52:19.306185 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.306166 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 20:52:19.308529 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.308503 2562 fs.go:135] Filesystem UUIDs: map[1cdcf8f2-99c9-41b5-a048-0dfa3c97fcb4:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cbec0781-bb78-49f2-a880-a48634574e0f:/dev/nvme0n1p4] Sep 09 20:52:19.308609 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.308527 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 09 20:52:19.315590 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.315427 2562 manager.go:216] Machine: {Timestamp:2026-09-09 20:52:19.313890766 +0000 UTC m=+0.355890567 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101339 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2799ea08f873dd7547eecab4345112 SystemUUID:ec2799ea-08f8-73dd-7547-eecab4345112 BootID:b5163b01-c461-442d-be44-b67d9542cad0 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:05:92:71:2e:41 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:05:92:71:2e:41 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:c5:71:27:b4:62 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 09 20:52:19.315590 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.315575 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 09 20:52:19.315744 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.315700 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 09 20:52:19.320847 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.320807 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 09 20:52:19.321016 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.320852 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-126.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 09 20:52:19.321693 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.321682 2562 topology_manager.go:138] "Creating topology manager with none policy" Sep 09 20:52:19.321734 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.321695 2562 container_manager_linux.go:309] "Creating device plugin manager" Sep 09 20:52:19.321734 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.321703 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 20:52:19.322420 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.322409 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 20:52:19.323179 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.323168 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 09 20:52:19.326014 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.326001 2562 state_mem.go:36] "Initialized new in-memory state store" Sep 09 20:52:19.326133 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.326124 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 09 20:52:19.326170 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.326154 2562 kubelet.go:486] "Attempting to sync node with API server" Sep 09 20:52:19.326170 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.326164 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 09 20:52:19.326227 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.326180 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 09 20:52:19.326271 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.326238 2562 kubelet.go:398] "Adding apiserver pod source" Sep 09 20:52:19.326316 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.326271 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 09 20:52:19.326695 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.326682 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 20:52:19.326734 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.326703 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 20:52:19.329445 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.329423 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9qz42" Sep 09 20:52:19.329708 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.329692 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 09 20:52:19.332243 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.332217 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 09 20:52:19.332313 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.332266 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 09 20:52:19.333335 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.333322 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 09 20:52:19.333374 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.333342 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 09 20:52:19.333374 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.333352 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 09 20:52:19.333374 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.333360 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 09 20:52:19.333374 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.333371 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 09 20:52:19.333497 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.333378 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 09 20:52:19.333497 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.333397 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 09 20:52:19.333497 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.333406 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 09 20:52:19.333497 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.333416 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 09 20:52:19.333497 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.333424 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 09 20:52:19.333497 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.333435 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 09 20:52:19.333497 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.333443 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 09 20:52:19.334994 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.334980 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 09 20:52:19.334994 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.334994 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 09 20:52:19.338240 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.338218 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-126.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 09 20:52:19.338356 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:19.338306 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-126.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 09 20:52:19.338356 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:19.338306 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 09 20:52:19.338817 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.338799 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9qz42" Sep 09 20:52:19.339811 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.339795 2562 server.go:1268] "Started kubelet" Sep 09 20:52:19.339964 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.339897 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 09 20:52:19.340007 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.339989 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 09 20:52:19.340071 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.340045 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 09 20:52:19.340893 ip-10-0-136-126 systemd[1]: Started Kubernetes Kubelet. Sep 09 20:52:19.341654 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.341457 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 09 20:52:19.342853 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.342838 2562 server.go:310] "Adding debug handlers to kubelet server" Sep 09 20:52:19.345649 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:19.345443 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 09 20:52:19.348557 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:19.347380 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-126.ec2.internal.18d3c2532416b16f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-126.ec2.internal,UID:ip-10-0-136-126.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-126.ec2.internal,},FirstTimestamp:2026-09-09 20:52:19.339759983 +0000 UTC m=+0.381759792,LastTimestamp:2026-09-09 20:52:19.339759983 +0000 UTC m=+0.381759792,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-126.ec2.internal,}" Sep 09 20:52:19.349645 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.349631 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 09 20:52:19.350070 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.350051 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 09 20:52:19.350642 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:19.350259 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Sep 09 20:52:19.350796 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.350776 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 09 20:52:19.350869 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.350798 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 09 20:52:19.350869 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.350866 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 09 20:52:19.350992 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.350907 2562 reconstruct.go:97] "Volume reconstruction finished" Sep 09 20:52:19.350992 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.350963 2562 reconciler.go:29] "Reconciler: start to sync state" Sep 09 20:52:19.351782 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.351769 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 09 20:52:19.351883 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.351873 2562 factory.go:55] Registering systemd factory Sep 09 20:52:19.351995 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.351986 2562 factory.go:223] Registration of the systemd container factory successfully Sep 09 20:52:19.352330 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.352313 2562 factory.go:153] Registering CRI-O factory Sep 09 20:52:19.352330 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.352334 2562 factory.go:223] Registration of the crio container factory successfully Sep 09 20:52:19.352450 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.352352 2562 factory.go:103] Registering Raw factory Sep 09 20:52:19.352450 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.352365 2562 manager.go:1196] Started watching for new ooms in manager Sep 09 20:52:19.352789 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.352775 2562 manager.go:349] Starting recovery of all containers Sep 09 20:52:19.362424 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.362256 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 20:52:19.365275 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:19.365238 2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-126.ec2.internal\" not found" node="ip-10-0-136-126.ec2.internal" Sep 09 20:52:19.366211 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.366193 2562 manager.go:354] Recovery completed Sep 09 20:52:19.371282 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.371268 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 20:52:19.374474 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.374452 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientMemory" Sep 09 20:52:19.374567 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.374489 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 20:52:19.374567 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.374499 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientPID" Sep 09 20:52:19.375089 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.375074 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 09 20:52:19.375185 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.375089 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 09 20:52:19.375185 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.375112 2562 state_mem.go:36] "Initialized new in-memory state store" Sep 09 20:52:19.375185 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.375076 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 20:52:19.375985 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.375816 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientMemory" Sep 09 20:52:19.376066 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.376004 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 20:52:19.376066 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.376023 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientPID" Sep 09 20:52:19.376383 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.376366 2562 policy_none.go:49] "None policy: Start" Sep 09 20:52:19.376428 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.376391 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 09 20:52:19.376428 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.376408 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 09 20:52:19.377437 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.377424 2562 policy_none.go:47] "Start" Sep 09 20:52:19.421723 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.421687 2562 manager.go:337] "Starting Device Plugin manager" Sep 09 20:52:19.425924 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:19.421759 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 09 20:52:19.425924 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.421773 2562 server.go:87] "Starting device plugin registration server" Sep 09 20:52:19.425924 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.422172 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 09 20:52:19.425924 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.422183 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 09 20:52:19.425924 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.422282 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 09 20:52:19.425924 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.422365 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 09 20:52:19.425924 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.422373 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 09 20:52:19.425924 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:19.423483 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 09 20:52:19.425924 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:19.423530 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-126.ec2.internal\" not found" Sep 09 20:52:19.482717 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.482680 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 09 20:52:19.483855 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.483836 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 09 20:52:19.483941 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.483907 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 09 20:52:19.483984 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.483951 2562 kubelet.go:2448] "Starting kubelet main sync loop" Sep 09 20:52:19.484074 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:19.484059 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 09 20:52:19.486641 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.486618 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 20:52:19.522877 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.522775 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 20:52:19.523907 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.523888 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientMemory" Sep 09 20:52:19.524029 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.523917 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 20:52:19.524029 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.523943 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientPID" Sep 09 20:52:19.524029 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.523970 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-126.ec2.internal" Sep 09 20:52:19.532441 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.532421 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-126.ec2.internal" Sep 09 20:52:19.532549 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:19.532451 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-126.ec2.internal\": node \"ip-10-0-136-126.ec2.internal\" not found" Sep 09 20:52:19.554233 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:19.554201 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Sep 09 20:52:19.584481 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.584442 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal"] Sep 09 20:52:19.584563 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.584545 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 20:52:19.585536 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.585517 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientMemory" Sep 09 20:52:19.585644 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.585551 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 20:52:19.585644 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.585565 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientPID" Sep 09 20:52:19.587895 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.587879 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 20:52:19.588015 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.587999 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" Sep 09 20:52:19.588071 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.588029 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 20:52:19.588800 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.588782 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientMemory" Sep 09 20:52:19.588874 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.588813 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 20:52:19.588874 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.588825 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientPID" Sep 09 20:52:19.588986 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.588822 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientMemory" Sep 09 20:52:19.588986 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.588919 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 20:52:19.588986 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.588952 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientPID" Sep 09 20:52:19.591249 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.591235 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Sep 09 20:52:19.591323 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.591262 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 20:52:19.592004 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.591990 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientMemory" Sep 09 20:52:19.592080 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.592015 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 20:52:19.592080 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.592027 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientPID" Sep 09 20:52:19.618742 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:19.618713 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-126.ec2.internal\" not found" node="ip-10-0-136-126.ec2.internal" Sep 09 20:52:19.623348 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:19.623331 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-126.ec2.internal\" not found" node="ip-10-0-136-126.ec2.internal" Sep 09 20:52:19.652132 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.652106 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/63346b7c1fd6eaae6b1974fddafa03ee-config\") pod \"kube-apiserver-proxy-ip-10-0-136-126.ec2.internal\" (UID: \"63346b7c1fd6eaae6b1974fddafa03ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" Sep 09 20:52:19.652298 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.652143 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3ad17323fa8ad3d6d259e0479cf3859-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal\" (UID: \"c3ad17323fa8ad3d6d259e0479cf3859\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Sep 09 20:52:19.652298 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.652162 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3ad17323fa8ad3d6d259e0479cf3859-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal\" (UID: \"c3ad17323fa8ad3d6d259e0479cf3859\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Sep 09 20:52:19.654545 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:19.654528 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Sep 09 20:52:19.753088 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.753053 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/63346b7c1fd6eaae6b1974fddafa03ee-config\") pod \"kube-apiserver-proxy-ip-10-0-136-126.ec2.internal\" (UID: \"63346b7c1fd6eaae6b1974fddafa03ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" Sep 09 20:52:19.753088 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.753086 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3ad17323fa8ad3d6d259e0479cf3859-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal\" (UID: \"c3ad17323fa8ad3d6d259e0479cf3859\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Sep 09 20:52:19.753304 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.753103 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3ad17323fa8ad3d6d259e0479cf3859-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal\" (UID: \"c3ad17323fa8ad3d6d259e0479cf3859\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Sep 09 20:52:19.753304 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.753149 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3ad17323fa8ad3d6d259e0479cf3859-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal\" (UID: \"c3ad17323fa8ad3d6d259e0479cf3859\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Sep 09 20:52:19.753304 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.753153 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/63346b7c1fd6eaae6b1974fddafa03ee-config\") pod \"kube-apiserver-proxy-ip-10-0-136-126.ec2.internal\" (UID: \"63346b7c1fd6eaae6b1974fddafa03ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" Sep 09 20:52:19.753304 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.753162 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3ad17323fa8ad3d6d259e0479cf3859-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal\" (UID: \"c3ad17323fa8ad3d6d259e0479cf3859\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Sep 09 20:52:19.755169 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:19.755152 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Sep 09 20:52:19.856074 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:19.855995 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Sep 09 20:52:19.921242 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.921207 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" Sep 09 20:52:19.927138 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:19.927038 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Sep 09 20:52:19.956754 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:19.956722 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Sep 09 20:52:20.057249 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:20.057214 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Sep 09 20:52:20.157800 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:20.157721 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Sep 09 20:52:20.186607 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.186581 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 20:52:20.251283 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.251250 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" Sep 09 20:52:20.265719 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.265690 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 20:52:20.267572 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.267558 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Sep 09 20:52:20.277809 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.277787 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 20:52:20.279956 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.279925 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 09 20:52:20.280088 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.280059 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 20:52:20.280203 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.280098 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 20:52:20.280203 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.280108 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 20:52:20.327273 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.327230 2562 apiserver.go:52] "Watching apiserver" Sep 09 20:52:20.341313 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.341286 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 09 20:52:20.343508 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.343461 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 20:47:19 +0000 UTC" deadline="2028-02-03 15:50:17.903493423 +0000 UTC" Sep 09 20:52:20.343508 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.343503 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12282h57m57.559992823s" Sep 09 20:52:20.343834 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.343810 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-69jvv","openshift-multus/multus-additional-cni-plugins-nptv6","openshift-multus/network-metrics-daemon-d6q8h","kube-system/global-pull-secret-syncer-kf998","kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal","openshift-cluster-node-tuning-operator/tuned-xktgp","openshift-dns/node-resolver-n7kn4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal","openshift-network-diagnostics/network-check-target-bbvk6","openshift-network-operator/iptables-alerter-ftj24","openshift-ovn-kubernetes/ovnkube-node-crjwd","kube-system/konnectivity-agent-gvw72","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk","openshift-image-registry/node-ca-5cx59"] Sep 09 20:52:20.346469 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.346448 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ftj24" Sep 09 20:52:20.348608 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.348589 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.350606 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.350586 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 09 20:52:20.350790 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.350776 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:20.350865 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:20.350834 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6q8h" podUID="ecd60cff-708d-4a89-827e-e01025163b5a" Sep 09 20:52:20.352070 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.352053 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 09 20:52:20.352131 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.352054 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 09 20:52:20.352202 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.352189 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 09 20:52:20.352273 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.352253 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 09 20:52:20.352311 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.352298 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 09 20:52:20.352311 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.352302 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 09 20:52:20.352383 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.352350 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h24zj\"" Sep 09 20:52:20.352416 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.352396 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 09 20:52:20.352543 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.352530 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 09 20:52:20.352781 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.352768 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m4s59\"" Sep 09 20:52:20.353106 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.353090 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5cx59" Sep 09 20:52:20.353186 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.353147 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 09 20:52:20.355241 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.355225 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gvw72" Sep 09 20:52:20.356570 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.356549 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-log-socket\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.356647 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.356576 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-host-cni-bin\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.356647 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.356592 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d55a4588-7960-4998-a99e-fca8f803b053-ovn-node-metrics-cert\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.356647 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.356612 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grkmm\" (UniqueName: \"kubernetes.io/projected/ecd60cff-708d-4a89-827e-e01025163b5a-kube-api-access-grkmm\") pod \"network-metrics-daemon-d6q8h\" (UID: \"ecd60cff-708d-4a89-827e-e01025163b5a\") " pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:20.356647 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.356630 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e4565ab-4f66-4753-b0d0-7d088a90d3da-host\") pod \"node-ca-5cx59\" (UID: \"4e4565ab-4f66-4753-b0d0-7d088a90d3da\") " pod="openshift-image-registry/node-ca-5cx59" Sep 09 20:52:20.356647 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.356646 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c30f7176-7e1b-40de-815d-8ad13d883631-iptables-alerter-script\") pod \"iptables-alerter-ftj24\" (UID: \"c30f7176-7e1b-40de-815d-8ad13d883631\") " pod="openshift-network-operator/iptables-alerter-ftj24" Sep 09 20:52:20.356822 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.356662 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbqsc\" (UniqueName: \"kubernetes.io/projected/c30f7176-7e1b-40de-815d-8ad13d883631-kube-api-access-jbqsc\") pod \"iptables-alerter-ftj24\" (UID: \"c30f7176-7e1b-40de-815d-8ad13d883631\") " pod="openshift-network-operator/iptables-alerter-ftj24" Sep 09 20:52:20.356822 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.356677 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-host-run-netns\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.356822 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.356691 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-etc-openvswitch\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.356822 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.356705 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-var-lib-openvswitch\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.356822 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.356720 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d55a4588-7960-4998-a99e-fca8f803b053-ovnkube-script-lib\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.356822 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.356746 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd60cff-708d-4a89-827e-e01025163b5a-metrics-certs\") pod \"network-metrics-daemon-d6q8h\" (UID: \"ecd60cff-708d-4a89-827e-e01025163b5a\") " pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:20.356822 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.356778 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56gvd\" (UniqueName: \"kubernetes.io/projected/4e4565ab-4f66-4753-b0d0-7d088a90d3da-kube-api-access-56gvd\") pod \"node-ca-5cx59\" (UID: \"4e4565ab-4f66-4753-b0d0-7d088a90d3da\") " pod="openshift-image-registry/node-ca-5cx59" Sep 09 20:52:20.356822 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.356804 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d55a4588-7960-4998-a99e-fca8f803b053-ovnkube-config\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.356822 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.356823 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4e4565ab-4f66-4753-b0d0-7d088a90d3da-serviceca\") pod \"node-ca-5cx59\" (UID: \"4e4565ab-4f66-4753-b0d0-7d088a90d3da\") " pod="openshift-image-registry/node-ca-5cx59" Sep 09 20:52:20.357105 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.356840 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c30f7176-7e1b-40de-815d-8ad13d883631-host-slash\") pod \"iptables-alerter-ftj24\" (UID: \"c30f7176-7e1b-40de-815d-8ad13d883631\") " pod="openshift-network-operator/iptables-alerter-ftj24" Sep 09 20:52:20.357105 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.356854 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-host-kubelet\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.357105 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.356868 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-systemd-units\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.357105 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.356893 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-run-openvswitch\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.357105 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.356908 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-node-log\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.357105 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.356923 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-host-run-ovn-kubernetes\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.357105 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.356954 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d55a4588-7960-4998-a99e-fca8f803b053-env-overrides\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.357105 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.356983 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-run-systemd\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.357105 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.357003 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-run-ovn\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.357105 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.357019 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-host-cni-netd\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.357105 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.357043 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-host-slash\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.357105 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.357061 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.357105 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.357077 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdw7r\" (UniqueName: \"kubernetes.io/projected/d55a4588-7960-4998-a99e-fca8f803b053-kube-api-access-bdw7r\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.357454 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.357430 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.357668 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.357655 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 09 20:52:20.357833 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.357817 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 09 20:52:20.357945 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.357918 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 09 20:52:20.358058 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.358038 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 09 20:52:20.358058 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.358042 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6k6gn\"" Sep 09 20:52:20.358201 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.358092 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 09 20:52:20.358265 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.358253 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mmtbs\"" Sep 09 20:52:20.359527 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.359513 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8n2qj\"" Sep 09 20:52:20.359712 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.359700 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 09 20:52:20.359787 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.359771 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:20.359864 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:20.359843 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbvk6" podUID="0495bf90-ab82-4771-b946-a12be5bef1af" Sep 09 20:52:20.359962 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.359949 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 09 20:52:20.362184 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.362158 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" Sep 09 20:52:20.364502 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.364482 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 09 20:52:20.364502 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.364498 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 09 20:52:20.364706 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.364516 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kf52q\"" Sep 09 20:52:20.364706 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.364546 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 09 20:52:20.364706 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.364610 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nptv6" Sep 09 20:52:20.365359 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.365063 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 20:52:20.368387 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.368131 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.368744 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.368579 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 09 20:52:20.368744 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.368632 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d6fcb\"" Sep 09 20:52:20.368744 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.368635 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 09 20:52:20.369072 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.369031 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 09 20:52:20.369072 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.369054 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 09 20:52:20.369072 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.369067 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 09 20:52:20.370605 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.370587 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 09 20:52:20.370795 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.370782 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n7kn4" Sep 09 20:52:20.371140 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.371126 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qlc2g\"" Sep 09 20:52:20.372737 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.372717 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 09 20:52:20.372868 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.372853 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 09 20:52:20.373099 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.373080 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:20.373180 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.373138 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hvcgc\"" Sep 09 20:52:20.373180 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:20.373145 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf998" podUID="0b2c736d-7746-4285-93bc-5ffd114396f1" Sep 09 20:52:20.405881 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.405846 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-44spl" Sep 09 20:52:20.416748 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.416726 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-44spl" Sep 09 20:52:20.452192 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.452165 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 09 20:52:20.458224 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458190 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-host-kubelet\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.458376 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458226 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-systemd-units\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.458376 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458257 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-node-log\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.458376 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458281 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-host-run-ovn-kubernetes\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.458376 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458309 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a88d1a84-874b-464b-b9c7-8952100b363e-agent-certs\") pod \"konnectivity-agent-gvw72\" (UID: \"a88d1a84-874b-464b-b9c7-8952100b363e\") " pod="kube-system/konnectivity-agent-gvw72" Sep 09 20:52:20.458376 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458316 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-systemd-units\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.458376 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458333 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-etc-modprobe-d\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.458376 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458308 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-host-kubelet\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.458376 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458376 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h8lp\" (UniqueName: \"kubernetes.io/projected/d2f2d9c9-c216-491f-b3e9-dac3643cd6db-kube-api-access-5h8lp\") pod \"multus-additional-cni-plugins-nptv6\" (UID: \"d2f2d9c9-c216-491f-b3e9-dac3643cd6db\") " pod="openshift-multus/multus-additional-cni-plugins-nptv6" Sep 09 20:52:20.458704 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458367 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-node-log\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.458704 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458389 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-host-run-ovn-kubernetes\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.458704 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458409 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-run-ovn\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.458704 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458431 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-host-cni-netd\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.458704 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458455 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e6c7fc2-6de9-465f-aa30-47bfa17c6b04-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gsvpk\" (UID: \"1e6c7fc2-6de9-465f-aa30-47bfa17c6b04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" Sep 09 20:52:20.458704 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458477 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-etc-sysconfig\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.458704 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458506 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-host-cni-netd\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.458704 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458518 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-etc-sysctl-d\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.458704 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458547 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-run\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.458704 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458506 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-run-ovn\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.458704 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458572 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36c1a56e-45e1-4724-811b-c32706894bda-tmp\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.458704 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458606 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2f2d9c9-c216-491f-b3e9-dac3643cd6db-cnibin\") pod \"multus-additional-cni-plugins-nptv6\" (UID: \"d2f2d9c9-c216-491f-b3e9-dac3643cd6db\") " pod="openshift-multus/multus-additional-cni-plugins-nptv6" Sep 09 20:52:20.458704 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458624 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-host-slash\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.458704 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458641 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.458704 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458667 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e6c7fc2-6de9-465f-aa30-47bfa17c6b04-device-dir\") pod \"aws-ebs-csi-driver-node-gsvpk\" (UID: \"1e6c7fc2-6de9-465f-aa30-47bfa17c6b04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" Sep 09 20:52:20.458704 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458683 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-etc-kubernetes\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.458704 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458690 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-host-slash\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.459504 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458710 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.459504 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458716 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-lib-modules\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.459504 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458745 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-var-lib-kubelet\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.459504 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458768 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/36c1a56e-45e1-4724-811b-c32706894bda-etc-tuned\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.459504 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458792 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgvjx\" (UniqueName: \"kubernetes.io/projected/36c1a56e-45e1-4724-811b-c32706894bda-kube-api-access-zgvjx\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.459504 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458817 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-log-socket\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.459504 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458841 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e6c7fc2-6de9-465f-aa30-47bfa17c6b04-sys-fs\") pod \"aws-ebs-csi-driver-node-gsvpk\" (UID: \"1e6c7fc2-6de9-465f-aa30-47bfa17c6b04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" Sep 09 20:52:20.459504 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458879 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk5vd\" (UniqueName: \"kubernetes.io/projected/0495bf90-ab82-4771-b946-a12be5bef1af-kube-api-access-rk5vd\") pod \"network-check-target-bbvk6\" (UID: \"0495bf90-ab82-4771-b946-a12be5bef1af\") " pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:20.459504 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458877 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-log-socket\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.459504 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458902 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-system-cni-dir\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.459504 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458948 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-multus-conf-dir\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.459504 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.458977 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grkmm\" (UniqueName: \"kubernetes.io/projected/ecd60cff-708d-4a89-827e-e01025163b5a-kube-api-access-grkmm\") pod \"network-metrics-daemon-d6q8h\" (UID: \"ecd60cff-708d-4a89-827e-e01025163b5a\") " pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:20.459504 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459007 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c46d28d-4f9c-4307-827a-9c9a31213cb2-tmp-dir\") pod \"node-resolver-n7kn4\" (UID: \"7c46d28d-4f9c-4307-827a-9c9a31213cb2\") " pod="openshift-dns/node-resolver-n7kn4" Sep 09 20:52:20.459504 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459077 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c30f7176-7e1b-40de-815d-8ad13d883631-iptables-alerter-script\") pod \"iptables-alerter-ftj24\" (UID: \"c30f7176-7e1b-40de-815d-8ad13d883631\") " pod="openshift-network-operator/iptables-alerter-ftj24" Sep 09 20:52:20.459504 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459121 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbqsc\" (UniqueName: \"kubernetes.io/projected/c30f7176-7e1b-40de-815d-8ad13d883631-kube-api-access-jbqsc\") pod \"iptables-alerter-ftj24\" (UID: \"c30f7176-7e1b-40de-815d-8ad13d883631\") " pod="openshift-network-operator/iptables-alerter-ftj24" Sep 09 20:52:20.459504 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459146 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-host-run-netns\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.459504 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459176 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-etc-openvswitch\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.460324 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459227 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a88d1a84-874b-464b-b9c7-8952100b363e-konnectivity-ca\") pod \"konnectivity-agent-gvw72\" (UID: \"a88d1a84-874b-464b-b9c7-8952100b363e\") " pod="kube-system/konnectivity-agent-gvw72" Sep 09 20:52:20.460324 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459250 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7htx7\" (UniqueName: \"kubernetes.io/projected/1e6c7fc2-6de9-465f-aa30-47bfa17c6b04-kube-api-access-7htx7\") pod \"aws-ebs-csi-driver-node-gsvpk\" (UID: \"1e6c7fc2-6de9-465f-aa30-47bfa17c6b04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" Sep 09 20:52:20.460324 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459272 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-host\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.460324 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459317 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2f2d9c9-c216-491f-b3e9-dac3643cd6db-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nptv6\" (UID: \"d2f2d9c9-c216-491f-b3e9-dac3643cd6db\") " pod="openshift-multus/multus-additional-cni-plugins-nptv6" Sep 09 20:52:20.460324 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459329 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-etc-openvswitch\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.460324 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459348 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d55a4588-7960-4998-a99e-fca8f803b053-ovnkube-script-lib\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.460324 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459375 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56gvd\" (UniqueName: \"kubernetes.io/projected/4e4565ab-4f66-4753-b0d0-7d088a90d3da-kube-api-access-56gvd\") pod \"node-ca-5cx59\" (UID: \"4e4565ab-4f66-4753-b0d0-7d088a90d3da\") " pod="openshift-image-registry/node-ca-5cx59" Sep 09 20:52:20.460324 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459519 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b2c736d-7746-4285-93bc-5ffd114396f1-dbus\") pod \"global-pull-secret-syncer-kf998\" (UID: \"0b2c736d-7746-4285-93bc-5ffd114396f1\") " pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:20.460324 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459549 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d2f2d9c9-c216-491f-b3e9-dac3643cd6db-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nptv6\" (UID: \"d2f2d9c9-c216-491f-b3e9-dac3643cd6db\") " pod="openshift-multus/multus-additional-cni-plugins-nptv6" Sep 09 20:52:20.460324 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459575 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-cni-binary-copy\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.460324 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459607 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-etc-kubernetes\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.460324 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459633 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d55a4588-7960-4998-a99e-fca8f803b053-ovnkube-config\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.460324 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459662 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-host-var-lib-cni-bin\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.460324 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459730 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c30f7176-7e1b-40de-815d-8ad13d883631-host-slash\") pod \"iptables-alerter-ftj24\" (UID: \"c30f7176-7e1b-40de-815d-8ad13d883631\") " pod="openshift-network-operator/iptables-alerter-ftj24" Sep 09 20:52:20.460324 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459765 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-run-openvswitch\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.460324 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459789 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d55a4588-7960-4998-a99e-fca8f803b053-env-overrides\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.460324 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459838 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c30f7176-7e1b-40de-815d-8ad13d883631-host-slash\") pod \"iptables-alerter-ftj24\" (UID: \"c30f7176-7e1b-40de-815d-8ad13d883631\") " pod="openshift-network-operator/iptables-alerter-ftj24" Sep 09 20:52:20.461194 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459885 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-etc-sysctl-conf\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.461194 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459897 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-host-run-netns\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.461194 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459908 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-host-run-netns\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.461194 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459951 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-run-systemd\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.461194 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459960 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-run-openvswitch\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.461194 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.459976 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e6c7fc2-6de9-465f-aa30-47bfa17c6b04-socket-dir\") pod \"aws-ebs-csi-driver-node-gsvpk\" (UID: \"1e6c7fc2-6de9-465f-aa30-47bfa17c6b04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" Sep 09 20:52:20.461194 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460000 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e6c7fc2-6de9-465f-aa30-47bfa17c6b04-registration-dir\") pod \"aws-ebs-csi-driver-node-gsvpk\" (UID: \"1e6c7fc2-6de9-465f-aa30-47bfa17c6b04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" Sep 09 20:52:20.461194 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460018 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-run-systemd\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.461194 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460022 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-sys\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.461194 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460047 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2f2d9c9-c216-491f-b3e9-dac3643cd6db-cni-binary-copy\") pod \"multus-additional-cni-plugins-nptv6\" (UID: \"d2f2d9c9-c216-491f-b3e9-dac3643cd6db\") " pod="openshift-multus/multus-additional-cni-plugins-nptv6" Sep 09 20:52:20.461194 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460079 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d2f2d9c9-c216-491f-b3e9-dac3643cd6db-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nptv6\" (UID: \"d2f2d9c9-c216-491f-b3e9-dac3643cd6db\") " pod="openshift-multus/multus-additional-cni-plugins-nptv6" Sep 09 20:52:20.461194 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460129 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-host-var-lib-cni-multus\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.461194 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460179 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdw7r\" (UniqueName: \"kubernetes.io/projected/d55a4588-7960-4998-a99e-fca8f803b053-kube-api-access-bdw7r\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.461194 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460233 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-cnibin\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.461194 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460251 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d55a4588-7960-4998-a99e-fca8f803b053-ovnkube-script-lib\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.461194 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460291 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d55a4588-7960-4998-a99e-fca8f803b053-ovnkube-config\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.461194 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460332 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-host-run-k8s-cni-cncf-io\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.461708 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460405 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pczzh\" (UniqueName: \"kubernetes.io/projected/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-kube-api-access-pczzh\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.461708 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460450 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-host-cni-bin\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.461708 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460526 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d55a4588-7960-4998-a99e-fca8f803b053-ovn-node-metrics-cert\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.461708 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460553 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-multus-daemon-config\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.461708 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460559 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-host-cni-bin\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.461708 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460577 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c30f7176-7e1b-40de-815d-8ad13d883631-iptables-alerter-script\") pod \"iptables-alerter-ftj24\" (UID: \"c30f7176-7e1b-40de-815d-8ad13d883631\") " pod="openshift-network-operator/iptables-alerter-ftj24" Sep 09 20:52:20.461708 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460579 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42gqw\" (UniqueName: \"kubernetes.io/projected/7c46d28d-4f9c-4307-827a-9c9a31213cb2-kube-api-access-42gqw\") pod \"node-resolver-n7kn4\" (UID: \"7c46d28d-4f9c-4307-827a-9c9a31213cb2\") " pod="openshift-dns/node-resolver-n7kn4" Sep 09 20:52:20.461708 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460631 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e4565ab-4f66-4753-b0d0-7d088a90d3da-host\") pod \"node-ca-5cx59\" (UID: \"4e4565ab-4f66-4753-b0d0-7d088a90d3da\") " pod="openshift-image-registry/node-ca-5cx59" Sep 09 20:52:20.461708 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460659 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b2c736d-7746-4285-93bc-5ffd114396f1-original-pull-secret\") pod \"global-pull-secret-syncer-kf998\" (UID: \"0b2c736d-7746-4285-93bc-5ffd114396f1\") " pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:20.461708 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460688 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2f2d9c9-c216-491f-b3e9-dac3643cd6db-system-cni-dir\") pod \"multus-additional-cni-plugins-nptv6\" (UID: \"d2f2d9c9-c216-491f-b3e9-dac3643cd6db\") " pod="openshift-multus/multus-additional-cni-plugins-nptv6" Sep 09 20:52:20.461708 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460692 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e4565ab-4f66-4753-b0d0-7d088a90d3da-host\") pod \"node-ca-5cx59\" (UID: \"4e4565ab-4f66-4753-b0d0-7d088a90d3da\") " pod="openshift-image-registry/node-ca-5cx59" Sep 09 20:52:20.461708 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460706 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d55a4588-7960-4998-a99e-fca8f803b053-env-overrides\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.461708 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460723 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-multus-socket-dir-parent\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.461708 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460746 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b2c736d-7746-4285-93bc-5ffd114396f1-kubelet-config\") pod \"global-pull-secret-syncer-kf998\" (UID: \"0b2c736d-7746-4285-93bc-5ffd114396f1\") " pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:20.461708 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460769 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-etc-systemd\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.461708 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460792 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-host-var-lib-kubelet\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.461708 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460819 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-var-lib-openvswitch\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.462239 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460862 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55a4588-7960-4998-a99e-fca8f803b053-var-lib-openvswitch\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.462239 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460871 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd60cff-708d-4a89-827e-e01025163b5a-metrics-certs\") pod \"network-metrics-daemon-d6q8h\" (UID: \"ecd60cff-708d-4a89-827e-e01025163b5a\") " pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:20.462239 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460899 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-multus-cni-dir\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.462239 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460897 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 09 20:52:20.462239 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.460945 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-host-run-multus-certs\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.462239 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:20.460991 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:52:20.462239 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.461007 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4e4565ab-4f66-4753-b0d0-7d088a90d3da-serviceca\") pod \"node-ca-5cx59\" (UID: \"4e4565ab-4f66-4753-b0d0-7d088a90d3da\") " pod="openshift-image-registry/node-ca-5cx59" Sep 09 20:52:20.462239 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.461032 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e6c7fc2-6de9-465f-aa30-47bfa17c6b04-etc-selinux\") pod \"aws-ebs-csi-driver-node-gsvpk\" (UID: \"1e6c7fc2-6de9-465f-aa30-47bfa17c6b04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" Sep 09 20:52:20.462239 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.461055 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-hostroot\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.462239 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:20.461081 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecd60cff-708d-4a89-827e-e01025163b5a-metrics-certs podName:ecd60cff-708d-4a89-827e-e01025163b5a nodeName:}" failed. No retries permitted until 2026-09-09 20:52:20.961059859 +0000 UTC m=+2.003059660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecd60cff-708d-4a89-827e-e01025163b5a-metrics-certs") pod "network-metrics-daemon-d6q8h" (UID: "ecd60cff-708d-4a89-827e-e01025163b5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:52:20.462239 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.461114 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7c46d28d-4f9c-4307-827a-9c9a31213cb2-hosts-file\") pod \"node-resolver-n7kn4\" (UID: \"7c46d28d-4f9c-4307-827a-9c9a31213cb2\") " pod="openshift-dns/node-resolver-n7kn4" Sep 09 20:52:20.462239 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.461346 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4e4565ab-4f66-4753-b0d0-7d088a90d3da-serviceca\") pod \"node-ca-5cx59\" (UID: \"4e4565ab-4f66-4753-b0d0-7d088a90d3da\") " pod="openshift-image-registry/node-ca-5cx59" Sep 09 20:52:20.464276 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.464253 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d55a4588-7960-4998-a99e-fca8f803b053-ovn-node-metrics-cert\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.470023 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.469996 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdw7r\" (UniqueName: \"kubernetes.io/projected/d55a4588-7960-4998-a99e-fca8f803b053-kube-api-access-bdw7r\") pod \"ovnkube-node-crjwd\" (UID: \"d55a4588-7960-4998-a99e-fca8f803b053\") " pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.472690 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.472668 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbqsc\" (UniqueName: \"kubernetes.io/projected/c30f7176-7e1b-40de-815d-8ad13d883631-kube-api-access-jbqsc\") pod \"iptables-alerter-ftj24\" (UID: \"c30f7176-7e1b-40de-815d-8ad13d883631\") " pod="openshift-network-operator/iptables-alerter-ftj24" Sep 09 20:52:20.472819 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.472749 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grkmm\" (UniqueName: \"kubernetes.io/projected/ecd60cff-708d-4a89-827e-e01025163b5a-kube-api-access-grkmm\") pod \"network-metrics-daemon-d6q8h\" (UID: \"ecd60cff-708d-4a89-827e-e01025163b5a\") " pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:20.474128 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.474107 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56gvd\" (UniqueName: \"kubernetes.io/projected/4e4565ab-4f66-4753-b0d0-7d088a90d3da-kube-api-access-56gvd\") pod \"node-ca-5cx59\" (UID: \"4e4565ab-4f66-4753-b0d0-7d088a90d3da\") " pod="openshift-image-registry/node-ca-5cx59" Sep 09 20:52:20.561859 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.561822 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgvjx\" (UniqueName: \"kubernetes.io/projected/36c1a56e-45e1-4724-811b-c32706894bda-kube-api-access-zgvjx\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.561859 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.561861 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e6c7fc2-6de9-465f-aa30-47bfa17c6b04-sys-fs\") pod \"aws-ebs-csi-driver-node-gsvpk\" (UID: \"1e6c7fc2-6de9-465f-aa30-47bfa17c6b04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" Sep 09 20:52:20.562180 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.561876 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk5vd\" (UniqueName: \"kubernetes.io/projected/0495bf90-ab82-4771-b946-a12be5bef1af-kube-api-access-rk5vd\") pod \"network-check-target-bbvk6\" (UID: \"0495bf90-ab82-4771-b946-a12be5bef1af\") " pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:20.562180 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.561892 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-system-cni-dir\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.562180 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.561910 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-multus-conf-dir\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.562180 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.561947 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c46d28d-4f9c-4307-827a-9c9a31213cb2-tmp-dir\") pod \"node-resolver-n7kn4\" (UID: \"7c46d28d-4f9c-4307-827a-9c9a31213cb2\") " pod="openshift-dns/node-resolver-n7kn4" Sep 09 20:52:20.562180 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.561975 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a88d1a84-874b-464b-b9c7-8952100b363e-konnectivity-ca\") pod \"konnectivity-agent-gvw72\" (UID: \"a88d1a84-874b-464b-b9c7-8952100b363e\") " pod="kube-system/konnectivity-agent-gvw72" Sep 09 20:52:20.562180 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.561988 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e6c7fc2-6de9-465f-aa30-47bfa17c6b04-sys-fs\") pod \"aws-ebs-csi-driver-node-gsvpk\" (UID: \"1e6c7fc2-6de9-465f-aa30-47bfa17c6b04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" Sep 09 20:52:20.562180 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.561998 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7htx7\" (UniqueName: \"kubernetes.io/projected/1e6c7fc2-6de9-465f-aa30-47bfa17c6b04-kube-api-access-7htx7\") pod \"aws-ebs-csi-driver-node-gsvpk\" (UID: \"1e6c7fc2-6de9-465f-aa30-47bfa17c6b04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" Sep 09 20:52:20.562180 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562049 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-host\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.562180 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562055 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-system-cni-dir\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.562180 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562076 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2f2d9c9-c216-491f-b3e9-dac3643cd6db-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nptv6\" (UID: \"d2f2d9c9-c216-491f-b3e9-dac3643cd6db\") " pod="openshift-multus/multus-additional-cni-plugins-nptv6" Sep 09 20:52:20.562180 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562103 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-multus-conf-dir\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.562180 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562107 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b2c736d-7746-4285-93bc-5ffd114396f1-dbus\") pod \"global-pull-secret-syncer-kf998\" (UID: \"0b2c736d-7746-4285-93bc-5ffd114396f1\") " pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:20.562180 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562120 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-host\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.562180 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562135 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d2f2d9c9-c216-491f-b3e9-dac3643cd6db-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nptv6\" (UID: \"d2f2d9c9-c216-491f-b3e9-dac3643cd6db\") " pod="openshift-multus/multus-additional-cni-plugins-nptv6" Sep 09 20:52:20.562180 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562154 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-cni-binary-copy\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.562180 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562160 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b2c736d-7746-4285-93bc-5ffd114396f1-dbus\") pod \"global-pull-secret-syncer-kf998\" (UID: \"0b2c736d-7746-4285-93bc-5ffd114396f1\") " pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:20.562180 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562171 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-etc-kubernetes\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.562180 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562191 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-host-var-lib-cni-bin\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.563035 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562208 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-etc-sysctl-conf\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.563035 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562223 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-host-run-netns\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.563035 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562237 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e6c7fc2-6de9-465f-aa30-47bfa17c6b04-socket-dir\") pod \"aws-ebs-csi-driver-node-gsvpk\" (UID: \"1e6c7fc2-6de9-465f-aa30-47bfa17c6b04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" Sep 09 20:52:20.563035 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562254 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e6c7fc2-6de9-465f-aa30-47bfa17c6b04-registration-dir\") pod \"aws-ebs-csi-driver-node-gsvpk\" (UID: \"1e6c7fc2-6de9-465f-aa30-47bfa17c6b04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" Sep 09 20:52:20.563035 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562269 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-sys\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.563035 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562284 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2f2d9c9-c216-491f-b3e9-dac3643cd6db-cni-binary-copy\") pod \"multus-additional-cni-plugins-nptv6\" (UID: \"d2f2d9c9-c216-491f-b3e9-dac3643cd6db\") " pod="openshift-multus/multus-additional-cni-plugins-nptv6" Sep 09 20:52:20.563035 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562302 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d2f2d9c9-c216-491f-b3e9-dac3643cd6db-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nptv6\" (UID: \"d2f2d9c9-c216-491f-b3e9-dac3643cd6db\") " pod="openshift-multus/multus-additional-cni-plugins-nptv6" Sep 09 20:52:20.563035 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562317 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-host-var-lib-cni-multus\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.563035 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562331 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2f2d9c9-c216-491f-b3e9-dac3643cd6db-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nptv6\" (UID: \"d2f2d9c9-c216-491f-b3e9-dac3643cd6db\") " pod="openshift-multus/multus-additional-cni-plugins-nptv6" Sep 09 20:52:20.563035 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562366 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-cnibin\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.563035 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562390 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-host-run-k8s-cni-cncf-io\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.563035 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562408 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pczzh\" (UniqueName: \"kubernetes.io/projected/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-kube-api-access-pczzh\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.563035 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562424 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-multus-daemon-config\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.563035 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562441 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42gqw\" (UniqueName: \"kubernetes.io/projected/7c46d28d-4f9c-4307-827a-9c9a31213cb2-kube-api-access-42gqw\") pod \"node-resolver-n7kn4\" (UID: \"7c46d28d-4f9c-4307-827a-9c9a31213cb2\") " pod="openshift-dns/node-resolver-n7kn4" Sep 09 20:52:20.563035 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562450 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e6c7fc2-6de9-465f-aa30-47bfa17c6b04-socket-dir\") pod \"aws-ebs-csi-driver-node-gsvpk\" (UID: \"1e6c7fc2-6de9-465f-aa30-47bfa17c6b04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" Sep 09 20:52:20.563035 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562454 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c46d28d-4f9c-4307-827a-9c9a31213cb2-tmp-dir\") pod \"node-resolver-n7kn4\" (UID: \"7c46d28d-4f9c-4307-827a-9c9a31213cb2\") " pod="openshift-dns/node-resolver-n7kn4" Sep 09 20:52:20.563035 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562460 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b2c736d-7746-4285-93bc-5ffd114396f1-original-pull-secret\") pod \"global-pull-secret-syncer-kf998\" (UID: \"0b2c736d-7746-4285-93bc-5ffd114396f1\") " pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:20.563770 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562493 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-etc-kubernetes\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.563770 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562495 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2f2d9c9-c216-491f-b3e9-dac3643cd6db-system-cni-dir\") pod \"multus-additional-cni-plugins-nptv6\" (UID: \"d2f2d9c9-c216-491f-b3e9-dac3643cd6db\") " pod="openshift-multus/multus-additional-cni-plugins-nptv6" Sep 09 20:52:20.563770 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:20.562518 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 20:52:20.563770 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562528 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-multus-socket-dir-parent\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.563770 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562553 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b2c736d-7746-4285-93bc-5ffd114396f1-kubelet-config\") pod \"global-pull-secret-syncer-kf998\" (UID: \"0b2c736d-7746-4285-93bc-5ffd114396f1\") " pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:20.563770 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562555 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-host-var-lib-cni-bin\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.563770 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:20.562569 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2c736d-7746-4285-93bc-5ffd114396f1-original-pull-secret podName:0b2c736d-7746-4285-93bc-5ffd114396f1 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:21.062555995 +0000 UTC m=+2.104555781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b2c736d-7746-4285-93bc-5ffd114396f1-original-pull-secret") pod "global-pull-secret-syncer-kf998" (UID: "0b2c736d-7746-4285-93bc-5ffd114396f1") : object "kube-system"/"original-pull-secret" not registered Sep 09 20:52:20.563770 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562655 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e6c7fc2-6de9-465f-aa30-47bfa17c6b04-registration-dir\") pod \"aws-ebs-csi-driver-node-gsvpk\" (UID: \"1e6c7fc2-6de9-465f-aa30-47bfa17c6b04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" Sep 09 20:52:20.563770 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562666 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-cni-binary-copy\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.563770 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562681 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a88d1a84-874b-464b-b9c7-8952100b363e-konnectivity-ca\") pod \"konnectivity-agent-gvw72\" (UID: \"a88d1a84-874b-464b-b9c7-8952100b363e\") " pod="kube-system/konnectivity-agent-gvw72" Sep 09 20:52:20.563770 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562670 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-etc-sysctl-conf\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.563770 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562700 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-host-run-netns\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.563770 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562695 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2f2d9c9-c216-491f-b3e9-dac3643cd6db-system-cni-dir\") pod \"multus-additional-cni-plugins-nptv6\" (UID: \"d2f2d9c9-c216-491f-b3e9-dac3643cd6db\") " pod="openshift-multus/multus-additional-cni-plugins-nptv6" Sep 09 20:52:20.563770 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562730 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-sys\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.563770 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562746 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-multus-socket-dir-parent\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.563770 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562756 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b2c736d-7746-4285-93bc-5ffd114396f1-kubelet-config\") pod \"global-pull-secret-syncer-kf998\" (UID: \"0b2c736d-7746-4285-93bc-5ffd114396f1\") " pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:20.563770 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562806 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-cnibin\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.564344 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562847 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-host-run-k8s-cni-cncf-io\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.564344 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562901 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-etc-systemd\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.564344 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562944 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-host-var-lib-kubelet\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.564344 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562980 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-etc-systemd\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.564344 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.562983 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-multus-cni-dir\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.564344 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563020 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-host-run-multus-certs\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.564344 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563030 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-multus-cni-dir\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.564344 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563047 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e6c7fc2-6de9-465f-aa30-47bfa17c6b04-etc-selinux\") pod \"aws-ebs-csi-driver-node-gsvpk\" (UID: \"1e6c7fc2-6de9-465f-aa30-47bfa17c6b04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" Sep 09 20:52:20.564344 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563072 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-hostroot\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.564344 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563078 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2f2d9c9-c216-491f-b3e9-dac3643cd6db-cni-binary-copy\") pod \"multus-additional-cni-plugins-nptv6\" (UID: \"d2f2d9c9-c216-491f-b3e9-dac3643cd6db\") " pod="openshift-multus/multus-additional-cni-plugins-nptv6" Sep 09 20:52:20.564344 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563074 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-host-var-lib-kubelet\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.564344 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563110 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-host-run-multus-certs\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.564344 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563135 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-host-var-lib-cni-multus\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.564344 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563136 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7c46d28d-4f9c-4307-827a-9c9a31213cb2-hosts-file\") pod \"node-resolver-n7kn4\" (UID: \"7c46d28d-4f9c-4307-827a-9c9a31213cb2\") " pod="openshift-dns/node-resolver-n7kn4" Sep 09 20:52:20.564344 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563151 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e6c7fc2-6de9-465f-aa30-47bfa17c6b04-etc-selinux\") pod \"aws-ebs-csi-driver-node-gsvpk\" (UID: \"1e6c7fc2-6de9-465f-aa30-47bfa17c6b04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" Sep 09 20:52:20.564344 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563173 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7c46d28d-4f9c-4307-827a-9c9a31213cb2-hosts-file\") pod \"node-resolver-n7kn4\" (UID: \"7c46d28d-4f9c-4307-827a-9c9a31213cb2\") " pod="openshift-dns/node-resolver-n7kn4" Sep 09 20:52:20.564344 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563187 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-hostroot\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.564344 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563174 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a88d1a84-874b-464b-b9c7-8952100b363e-agent-certs\") pod \"konnectivity-agent-gvw72\" (UID: \"a88d1a84-874b-464b-b9c7-8952100b363e\") " pod="kube-system/konnectivity-agent-gvw72" Sep 09 20:52:20.565048 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563223 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-etc-modprobe-d\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.565048 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563252 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h8lp\" (UniqueName: \"kubernetes.io/projected/d2f2d9c9-c216-491f-b3e9-dac3643cd6db-kube-api-access-5h8lp\") pod \"multus-additional-cni-plugins-nptv6\" (UID: \"d2f2d9c9-c216-491f-b3e9-dac3643cd6db\") " pod="openshift-multus/multus-additional-cni-plugins-nptv6" Sep 09 20:52:20.565048 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563282 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e6c7fc2-6de9-465f-aa30-47bfa17c6b04-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gsvpk\" (UID: \"1e6c7fc2-6de9-465f-aa30-47bfa17c6b04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" Sep 09 20:52:20.565048 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563252 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d2f2d9c9-c216-491f-b3e9-dac3643cd6db-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nptv6\" (UID: \"d2f2d9c9-c216-491f-b3e9-dac3643cd6db\") " pod="openshift-multus/multus-additional-cni-plugins-nptv6" Sep 09 20:52:20.565048 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563306 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-etc-sysconfig\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.565048 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563335 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d2f2d9c9-c216-491f-b3e9-dac3643cd6db-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nptv6\" (UID: \"d2f2d9c9-c216-491f-b3e9-dac3643cd6db\") " pod="openshift-multus/multus-additional-cni-plugins-nptv6" Sep 09 20:52:20.565048 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563344 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-multus-daemon-config\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.565048 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563361 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-etc-sysconfig\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.565048 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563357 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-etc-sysctl-d\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.565048 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563371 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e6c7fc2-6de9-465f-aa30-47bfa17c6b04-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gsvpk\" (UID: \"1e6c7fc2-6de9-465f-aa30-47bfa17c6b04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" Sep 09 20:52:20.565048 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563342 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-etc-modprobe-d\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.565048 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563388 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-run\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.565048 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563416 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36c1a56e-45e1-4724-811b-c32706894bda-tmp\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.565048 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563428 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-run\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.565048 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563433 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2f2d9c9-c216-491f-b3e9-dac3643cd6db-cnibin\") pod \"multus-additional-cni-plugins-nptv6\" (UID: \"d2f2d9c9-c216-491f-b3e9-dac3643cd6db\") " pod="openshift-multus/multus-additional-cni-plugins-nptv6" Sep 09 20:52:20.565048 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563449 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-etc-sysctl-d\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.565048 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563461 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e6c7fc2-6de9-465f-aa30-47bfa17c6b04-device-dir\") pod \"aws-ebs-csi-driver-node-gsvpk\" (UID: \"1e6c7fc2-6de9-465f-aa30-47bfa17c6b04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" Sep 09 20:52:20.565679 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563484 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-etc-kubernetes\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.565679 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563507 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-lib-modules\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.565679 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563529 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-var-lib-kubelet\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.565679 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563541 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e6c7fc2-6de9-465f-aa30-47bfa17c6b04-device-dir\") pod \"aws-ebs-csi-driver-node-gsvpk\" (UID: \"1e6c7fc2-6de9-465f-aa30-47bfa17c6b04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" Sep 09 20:52:20.565679 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563553 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/36c1a56e-45e1-4724-811b-c32706894bda-etc-tuned\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.565679 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563580 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2f2d9c9-c216-491f-b3e9-dac3643cd6db-cnibin\") pod \"multus-additional-cni-plugins-nptv6\" (UID: \"d2f2d9c9-c216-491f-b3e9-dac3643cd6db\") " pod="openshift-multus/multus-additional-cni-plugins-nptv6" Sep 09 20:52:20.565679 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563578 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-etc-kubernetes\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.565679 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563631 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-var-lib-kubelet\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.565679 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.563659 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36c1a56e-45e1-4724-811b-c32706894bda-lib-modules\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.566103 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.566025 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/36c1a56e-45e1-4724-811b-c32706894bda-etc-tuned\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.566103 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.566077 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36c1a56e-45e1-4724-811b-c32706894bda-tmp\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.566335 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.566309 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a88d1a84-874b-464b-b9c7-8952100b363e-agent-certs\") pod \"konnectivity-agent-gvw72\" (UID: \"a88d1a84-874b-464b-b9c7-8952100b363e\") " pod="kube-system/konnectivity-agent-gvw72" Sep 09 20:52:20.569458 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:20.569413 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 20:52:20.569458 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:20.569439 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 20:52:20.569458 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:20.569453 2562 projected.go:196] Error preparing data for projected volume kube-api-access-rk5vd for pod openshift-network-diagnostics/network-check-target-bbvk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:52:20.569757 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:20.569529 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0495bf90-ab82-4771-b946-a12be5bef1af-kube-api-access-rk5vd podName:0495bf90-ab82-4771-b946-a12be5bef1af nodeName:}" failed. No retries permitted until 2026-09-09 20:52:21.069511754 +0000 UTC m=+2.111511542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rk5vd" (UniqueName: "kubernetes.io/projected/0495bf90-ab82-4771-b946-a12be5bef1af-kube-api-access-rk5vd") pod "network-check-target-bbvk6" (UID: "0495bf90-ab82-4771-b946-a12be5bef1af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:52:20.571882 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.571857 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7htx7\" (UniqueName: \"kubernetes.io/projected/1e6c7fc2-6de9-465f-aa30-47bfa17c6b04-kube-api-access-7htx7\") pod \"aws-ebs-csi-driver-node-gsvpk\" (UID: \"1e6c7fc2-6de9-465f-aa30-47bfa17c6b04\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" Sep 09 20:52:20.572294 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.572275 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgvjx\" (UniqueName: \"kubernetes.io/projected/36c1a56e-45e1-4724-811b-c32706894bda-kube-api-access-zgvjx\") pod \"tuned-xktgp\" (UID: \"36c1a56e-45e1-4724-811b-c32706894bda\") " pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.572351 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.572275 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pczzh\" (UniqueName: \"kubernetes.io/projected/346ba7a6-b7db-48ac-9dd8-5793933dd7cd-kube-api-access-pczzh\") pod \"multus-69jvv\" (UID: \"346ba7a6-b7db-48ac-9dd8-5793933dd7cd\") " pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.572351 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.572325 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42gqw\" (UniqueName: \"kubernetes.io/projected/7c46d28d-4f9c-4307-827a-9c9a31213cb2-kube-api-access-42gqw\") pod \"node-resolver-n7kn4\" (UID: \"7c46d28d-4f9c-4307-827a-9c9a31213cb2\") " pod="openshift-dns/node-resolver-n7kn4" Sep 09 20:52:20.572637 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.572619 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h8lp\" (UniqueName: \"kubernetes.io/projected/d2f2d9c9-c216-491f-b3e9-dac3643cd6db-kube-api-access-5h8lp\") pod \"multus-additional-cni-plugins-nptv6\" (UID: \"d2f2d9c9-c216-491f-b3e9-dac3643cd6db\") " pod="openshift-multus/multus-additional-cni-plugins-nptv6" Sep 09 20:52:20.636259 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:20.636207 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63346b7c1fd6eaae6b1974fddafa03ee.slice/crio-af7fd238983cdb95baf98e7cd8f357e3dcc91612434dd1be00f7484a98e60b01 WatchSource:0}: Error finding container af7fd238983cdb95baf98e7cd8f357e3dcc91612434dd1be00f7484a98e60b01: Status 404 returned error can't find the container with id af7fd238983cdb95baf98e7cd8f357e3dcc91612434dd1be00f7484a98e60b01 Sep 09 20:52:20.636588 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:20.636555 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ad17323fa8ad3d6d259e0479cf3859.slice/crio-b323644673f06447237ae0d91b997870cb82cd280e2a7ef1ccc192afa60593c1 WatchSource:0}: Error finding container b323644673f06447237ae0d91b997870cb82cd280e2a7ef1ccc192afa60593c1: Status 404 returned error can't find the container with id b323644673f06447237ae0d91b997870cb82cd280e2a7ef1ccc192afa60593c1 Sep 09 20:52:20.641443 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.641415 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 20:52:20.676872 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.676767 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ftj24" Sep 09 20:52:20.686408 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:20.686374 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30f7176_7e1b_40de_815d_8ad13d883631.slice/crio-51e5d39b5a36f3061ecc6b5f4670f066565f0865e515eed202530f0158ce54b0 WatchSource:0}: Error finding container 51e5d39b5a36f3061ecc6b5f4670f066565f0865e515eed202530f0158ce54b0: Status 404 returned error can't find the container with id 51e5d39b5a36f3061ecc6b5f4670f066565f0865e515eed202530f0158ce54b0 Sep 09 20:52:20.693745 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.693725 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:20.702852 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:20.702824 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd55a4588_7960_4998_a99e_fca8f803b053.slice/crio-adebafc870019a654727187966f7ee9d94fbb8eb441caddf3472d73acaf486df WatchSource:0}: Error finding container adebafc870019a654727187966f7ee9d94fbb8eb441caddf3472d73acaf486df: Status 404 returned error can't find the container with id adebafc870019a654727187966f7ee9d94fbb8eb441caddf3472d73acaf486df Sep 09 20:52:20.710541 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.710525 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5cx59" Sep 09 20:52:20.718609 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:20.718586 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4565ab_4f66_4753_b0d0_7d088a90d3da.slice/crio-4437ae01fcb3d89da595f16f879650cddcd2cf34ba058c93ea2c6e0b9278e569 WatchSource:0}: Error finding container 4437ae01fcb3d89da595f16f879650cddcd2cf34ba058c93ea2c6e0b9278e569: Status 404 returned error can't find the container with id 4437ae01fcb3d89da595f16f879650cddcd2cf34ba058c93ea2c6e0b9278e569 Sep 09 20:52:20.727429 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.727408 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gvw72" Sep 09 20:52:20.736070 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:20.736046 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88d1a84_874b_464b_b9c7_8952100b363e.slice/crio-25fbd6134acc70cc1609e9c6a534f8a07acbd8b56dd3287530e586a2179e712e WatchSource:0}: Error finding container 25fbd6134acc70cc1609e9c6a534f8a07acbd8b56dd3287530e586a2179e712e: Status 404 returned error can't find the container with id 25fbd6134acc70cc1609e9c6a534f8a07acbd8b56dd3287530e586a2179e712e Sep 09 20:52:20.742044 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.742025 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xktgp" Sep 09 20:52:20.750060 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:20.750036 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36c1a56e_45e1_4724_811b_c32706894bda.slice/crio-ebd6dd89d7b7ebd297bf03c0413cf957b60d37761dca660e9777469b199f8117 WatchSource:0}: Error finding container ebd6dd89d7b7ebd297bf03c0413cf957b60d37761dca660e9777469b199f8117: Status 404 returned error can't find the container with id ebd6dd89d7b7ebd297bf03c0413cf957b60d37761dca660e9777469b199f8117 Sep 09 20:52:20.756816 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.756798 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" Sep 09 20:52:20.764521 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:20.764496 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e6c7fc2_6de9_465f_aa30_47bfa17c6b04.slice/crio-cf94442ab05950845564aed8aa5f5f0ec44763cbd3105336afea3b4343edb757 WatchSource:0}: Error finding container cf94442ab05950845564aed8aa5f5f0ec44763cbd3105336afea3b4343edb757: Status 404 returned error can't find the container with id cf94442ab05950845564aed8aa5f5f0ec44763cbd3105336afea3b4343edb757 Sep 09 20:52:20.780059 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.780034 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nptv6" Sep 09 20:52:20.789155 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:20.789130 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f2d9c9_c216_491f_b3e9_dac3643cd6db.slice/crio-71a0f4fa2fb09b1a70e7e3ab1d22a14074bdafcf9689ec3531d8e20fc9c6ac8b WatchSource:0}: Error finding container 71a0f4fa2fb09b1a70e7e3ab1d22a14074bdafcf9689ec3531d8e20fc9c6ac8b: Status 404 returned error can't find the container with id 71a0f4fa2fb09b1a70e7e3ab1d22a14074bdafcf9689ec3531d8e20fc9c6ac8b Sep 09 20:52:20.793296 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.793277 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-69jvv" Sep 09 20:52:20.798194 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.798173 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n7kn4" Sep 09 20:52:20.801330 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:20.801310 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod346ba7a6_b7db_48ac_9dd8_5793933dd7cd.slice/crio-83addf8d62c85484f036566d3339e108126967ba6e76ababad47d266e9b55ebf WatchSource:0}: Error finding container 83addf8d62c85484f036566d3339e108126967ba6e76ababad47d266e9b55ebf: Status 404 returned error can't find the container with id 83addf8d62c85484f036566d3339e108126967ba6e76ababad47d266e9b55ebf Sep 09 20:52:20.806158 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:20.806132 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c46d28d_4f9c_4307_827a_9c9a31213cb2.slice/crio-f3d305e01bdc5698fe6c19717a4f3c19322f800bdbdef86d9d568727d3fe8404 WatchSource:0}: Error finding container f3d305e01bdc5698fe6c19717a4f3c19322f800bdbdef86d9d568727d3fe8404: Status 404 returned error can't find the container with id f3d305e01bdc5698fe6c19717a4f3c19322f800bdbdef86d9d568727d3fe8404 Sep 09 20:52:20.807849 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:20.807831 2562 kuberuntime_manager.go:1459] "Unhandled Error" err="container dns-node-resolver start failed in pod node-resolver-n7kn4_openshift-dns(7c46d28d-4f9c-4307-827a-9c9a31213cb2): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Sep 09 20:52:20.807899 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:20.807862 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dns-node-resolver\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-dns/node-resolver-n7kn4" podUID="7c46d28d-4f9c-4307-827a-9c9a31213cb2" Sep 09 20:52:20.886977 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.886925 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 20:52:20.970176 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:20.970075 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd60cff-708d-4a89-827e-e01025163b5a-metrics-certs\") pod \"network-metrics-daemon-d6q8h\" (UID: \"ecd60cff-708d-4a89-827e-e01025163b5a\") " pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:20.970678 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:20.970274 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:52:20.970742 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:20.970728 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecd60cff-708d-4a89-827e-e01025163b5a-metrics-certs podName:ecd60cff-708d-4a89-827e-e01025163b5a nodeName:}" failed. No retries permitted until 2026-09-09 20:52:21.97070885 +0000 UTC m=+3.012708656 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecd60cff-708d-4a89-827e-e01025163b5a-metrics-certs") pod "network-metrics-daemon-d6q8h" (UID: "ecd60cff-708d-4a89-827e-e01025163b5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:52:21.070943 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:21.070906 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b2c736d-7746-4285-93bc-5ffd114396f1-original-pull-secret\") pod \"global-pull-secret-syncer-kf998\" (UID: \"0b2c736d-7746-4285-93bc-5ffd114396f1\") " pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:21.071132 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:21.070997 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk5vd\" (UniqueName: \"kubernetes.io/projected/0495bf90-ab82-4771-b946-a12be5bef1af-kube-api-access-rk5vd\") pod \"network-check-target-bbvk6\" (UID: \"0495bf90-ab82-4771-b946-a12be5bef1af\") " pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:21.071132 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:21.071090 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 20:52:21.071247 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:21.071130 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 20:52:21.071247 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:21.071155 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2c736d-7746-4285-93bc-5ffd114396f1-original-pull-secret podName:0b2c736d-7746-4285-93bc-5ffd114396f1 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:22.071138163 +0000 UTC m=+3.113137955 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b2c736d-7746-4285-93bc-5ffd114396f1-original-pull-secret") pod "global-pull-secret-syncer-kf998" (UID: "0b2c736d-7746-4285-93bc-5ffd114396f1") : object "kube-system"/"original-pull-secret" not registered Sep 09 20:52:21.071247 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:21.071159 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 20:52:21.071247 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:21.071176 2562 projected.go:196] Error preparing data for projected volume kube-api-access-rk5vd for pod openshift-network-diagnostics/network-check-target-bbvk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:52:21.071247 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:21.071225 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0495bf90-ab82-4771-b946-a12be5bef1af-kube-api-access-rk5vd podName:0495bf90-ab82-4771-b946-a12be5bef1af nodeName:}" failed. No retries permitted until 2026-09-09 20:52:22.071210286 +0000 UTC m=+3.113210072 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rk5vd" (UniqueName: "kubernetes.io/projected/0495bf90-ab82-4771-b946-a12be5bef1af-kube-api-access-rk5vd") pod "network-check-target-bbvk6" (UID: "0495bf90-ab82-4771-b946-a12be5bef1af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:52:21.145821 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:21.145786 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 20:52:21.266086 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:21.266005 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 20:52:21.420116 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:21.419999 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:47:20 +0000 UTC" deadline="2028-05-17 10:16:17.553045694 +0000 UTC" Sep 09 20:52:21.420116 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:21.420115 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14773h23m56.132936584s" Sep 09 20:52:21.495583 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:21.495553 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:21.495760 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:21.495676 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbvk6" podUID="0495bf90-ab82-4771-b946-a12be5bef1af" Sep 09 20:52:21.498159 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:21.497535 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n7kn4" event={"ID":"7c46d28d-4f9c-4307-827a-9c9a31213cb2","Type":"ContainerStarted","Data":"f3d305e01bdc5698fe6c19717a4f3c19322f800bdbdef86d9d568727d3fe8404"} Sep 09 20:52:21.499750 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:21.499710 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dns-node-resolver\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:40bc1b9c6b608369978e9f165c5e22d7a75fd91b3480cba60cd275228d721616\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-dns/node-resolver-n7kn4" podUID="7c46d28d-4f9c-4307-827a-9c9a31213cb2" Sep 09 20:52:21.505057 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:21.505024 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-69jvv" event={"ID":"346ba7a6-b7db-48ac-9dd8-5793933dd7cd","Type":"ContainerStarted","Data":"83addf8d62c85484f036566d3339e108126967ba6e76ababad47d266e9b55ebf"} Sep 09 20:52:21.514730 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:21.514694 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gvw72" event={"ID":"a88d1a84-874b-464b-b9c7-8952100b363e","Type":"ContainerStarted","Data":"25fbd6134acc70cc1609e9c6a534f8a07acbd8b56dd3287530e586a2179e712e"} Sep 09 20:52:21.518615 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:21.518461 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ftj24" event={"ID":"c30f7176-7e1b-40de-815d-8ad13d883631","Type":"ContainerStarted","Data":"51e5d39b5a36f3061ecc6b5f4670f066565f0865e515eed202530f0158ce54b0"} Sep 09 20:52:21.522321 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:21.522232 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" event={"ID":"c3ad17323fa8ad3d6d259e0479cf3859","Type":"ContainerStarted","Data":"b323644673f06447237ae0d91b997870cb82cd280e2a7ef1ccc192afa60593c1"} Sep 09 20:52:21.526899 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:21.526836 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" event={"ID":"63346b7c1fd6eaae6b1974fddafa03ee","Type":"ContainerStarted","Data":"af7fd238983cdb95baf98e7cd8f357e3dcc91612434dd1be00f7484a98e60b01"} Sep 09 20:52:21.530345 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:21.530260 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nptv6" event={"ID":"d2f2d9c9-c216-491f-b3e9-dac3643cd6db","Type":"ContainerStarted","Data":"71a0f4fa2fb09b1a70e7e3ab1d22a14074bdafcf9689ec3531d8e20fc9c6ac8b"} Sep 09 20:52:21.533186 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:21.533155 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" event={"ID":"1e6c7fc2-6de9-465f-aa30-47bfa17c6b04","Type":"ContainerStarted","Data":"cf94442ab05950845564aed8aa5f5f0ec44763cbd3105336afea3b4343edb757"} Sep 09 20:52:21.538714 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:21.538682 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xktgp" event={"ID":"36c1a56e-45e1-4724-811b-c32706894bda","Type":"ContainerStarted","Data":"ebd6dd89d7b7ebd297bf03c0413cf957b60d37761dca660e9777469b199f8117"} Sep 09 20:52:21.542724 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:21.542659 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5cx59" event={"ID":"4e4565ab-4f66-4753-b0d0-7d088a90d3da","Type":"ContainerStarted","Data":"4437ae01fcb3d89da595f16f879650cddcd2cf34ba058c93ea2c6e0b9278e569"} Sep 09 20:52:21.545521 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:21.545485 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" event={"ID":"d55a4588-7960-4998-a99e-fca8f803b053","Type":"ContainerStarted","Data":"adebafc870019a654727187966f7ee9d94fbb8eb441caddf3472d73acaf486df"} Sep 09 20:52:21.984722 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:21.984627 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd60cff-708d-4a89-827e-e01025163b5a-metrics-certs\") pod \"network-metrics-daemon-d6q8h\" (UID: \"ecd60cff-708d-4a89-827e-e01025163b5a\") " pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:21.984918 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:21.984785 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:52:21.984918 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:21.984853 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecd60cff-708d-4a89-827e-e01025163b5a-metrics-certs podName:ecd60cff-708d-4a89-827e-e01025163b5a nodeName:}" failed. No retries permitted until 2026-09-09 20:52:23.984835525 +0000 UTC m=+5.026835329 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecd60cff-708d-4a89-827e-e01025163b5a-metrics-certs") pod "network-metrics-daemon-d6q8h" (UID: "ecd60cff-708d-4a89-827e-e01025163b5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:52:22.085667 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:22.085625 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk5vd\" (UniqueName: \"kubernetes.io/projected/0495bf90-ab82-4771-b946-a12be5bef1af-kube-api-access-rk5vd\") pod \"network-check-target-bbvk6\" (UID: \"0495bf90-ab82-4771-b946-a12be5bef1af\") " pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:22.085876 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:22.085706 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b2c736d-7746-4285-93bc-5ffd114396f1-original-pull-secret\") pod \"global-pull-secret-syncer-kf998\" (UID: \"0b2c736d-7746-4285-93bc-5ffd114396f1\") " pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:22.085876 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:22.085866 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 20:52:22.086032 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:22.085937 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2c736d-7746-4285-93bc-5ffd114396f1-original-pull-secret podName:0b2c736d-7746-4285-93bc-5ffd114396f1 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:24.085917008 +0000 UTC m=+5.127916822 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b2c736d-7746-4285-93bc-5ffd114396f1-original-pull-secret") pod "global-pull-secret-syncer-kf998" (UID: "0b2c736d-7746-4285-93bc-5ffd114396f1") : object "kube-system"/"original-pull-secret" not registered Sep 09 20:52:22.086431 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:22.086410 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 20:52:22.086496 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:22.086441 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 20:52:22.086496 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:22.086455 2562 projected.go:196] Error preparing data for projected volume kube-api-access-rk5vd for pod openshift-network-diagnostics/network-check-target-bbvk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:52:22.086596 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:22.086509 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0495bf90-ab82-4771-b946-a12be5bef1af-kube-api-access-rk5vd podName:0495bf90-ab82-4771-b946-a12be5bef1af nodeName:}" failed. No retries permitted until 2026-09-09 20:52:24.086489367 +0000 UTC m=+5.128489160 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rk5vd" (UniqueName: "kubernetes.io/projected/0495bf90-ab82-4771-b946-a12be5bef1af-kube-api-access-rk5vd") pod "network-check-target-bbvk6" (UID: "0495bf90-ab82-4771-b946-a12be5bef1af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:52:22.420744 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:22.420645 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:47:20 +0000 UTC" deadline="2028-04-06 23:46:18.593116105 +0000 UTC" Sep 09 20:52:22.420744 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:22.420691 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13802h53m56.172430897s" Sep 09 20:52:22.484405 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:22.484368 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:22.484603 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:22.484516 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf998" podUID="0b2c736d-7746-4285-93bc-5ffd114396f1" Sep 09 20:52:22.485027 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:22.484984 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:22.485197 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:22.485175 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6q8h" podUID="ecd60cff-708d-4a89-827e-e01025163b5a" Sep 09 20:52:22.874178 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:22.874089 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 20:52:23.488981 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:23.488481 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:23.488981 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:23.488618 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbvk6" podUID="0495bf90-ab82-4771-b946-a12be5bef1af" Sep 09 20:52:24.001843 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:24.001802 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd60cff-708d-4a89-827e-e01025163b5a-metrics-certs\") pod \"network-metrics-daemon-d6q8h\" (UID: \"ecd60cff-708d-4a89-827e-e01025163b5a\") " pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:24.002047 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:24.001924 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:52:24.002047 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:24.002023 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecd60cff-708d-4a89-827e-e01025163b5a-metrics-certs podName:ecd60cff-708d-4a89-827e-e01025163b5a nodeName:}" failed. No retries permitted until 2026-09-09 20:52:28.001988436 +0000 UTC m=+9.043988230 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecd60cff-708d-4a89-827e-e01025163b5a-metrics-certs") pod "network-metrics-daemon-d6q8h" (UID: "ecd60cff-708d-4a89-827e-e01025163b5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:52:24.103223 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:24.103184 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk5vd\" (UniqueName: \"kubernetes.io/projected/0495bf90-ab82-4771-b946-a12be5bef1af-kube-api-access-rk5vd\") pod \"network-check-target-bbvk6\" (UID: \"0495bf90-ab82-4771-b946-a12be5bef1af\") " pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:24.103405 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:24.103261 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b2c736d-7746-4285-93bc-5ffd114396f1-original-pull-secret\") pod \"global-pull-secret-syncer-kf998\" (UID: \"0b2c736d-7746-4285-93bc-5ffd114396f1\") " pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:24.103405 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:24.103380 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 20:52:24.103405 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:24.103383 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 20:52:24.103577 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:24.103416 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 20:52:24.103577 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:24.103431 2562 projected.go:196] Error preparing data for projected volume kube-api-access-rk5vd for pod openshift-network-diagnostics/network-check-target-bbvk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:52:24.103577 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:24.103443 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2c736d-7746-4285-93bc-5ffd114396f1-original-pull-secret podName:0b2c736d-7746-4285-93bc-5ffd114396f1 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:28.103428297 +0000 UTC m=+9.145428086 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b2c736d-7746-4285-93bc-5ffd114396f1-original-pull-secret") pod "global-pull-secret-syncer-kf998" (UID: "0b2c736d-7746-4285-93bc-5ffd114396f1") : object "kube-system"/"original-pull-secret" not registered Sep 09 20:52:24.103577 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:24.103488 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0495bf90-ab82-4771-b946-a12be5bef1af-kube-api-access-rk5vd podName:0495bf90-ab82-4771-b946-a12be5bef1af nodeName:}" failed. No retries permitted until 2026-09-09 20:52:28.103470752 +0000 UTC m=+9.145470557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rk5vd" (UniqueName: "kubernetes.io/projected/0495bf90-ab82-4771-b946-a12be5bef1af-kube-api-access-rk5vd") pod "network-check-target-bbvk6" (UID: "0495bf90-ab82-4771-b946-a12be5bef1af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:52:24.485072 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:24.484265 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:24.485072 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:24.484416 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf998" podUID="0b2c736d-7746-4285-93bc-5ffd114396f1" Sep 09 20:52:24.485072 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:24.484499 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:24.485072 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:24.484584 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6q8h" podUID="ecd60cff-708d-4a89-827e-e01025163b5a" Sep 09 20:52:25.484423 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:25.484383 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:25.484948 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:25.484538 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbvk6" podUID="0495bf90-ab82-4771-b946-a12be5bef1af" Sep 09 20:52:26.485286 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:26.484544 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:26.485286 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:26.484677 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf998" podUID="0b2c736d-7746-4285-93bc-5ffd114396f1" Sep 09 20:52:26.485286 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:26.485118 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:26.485286 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:26.485234 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6q8h" podUID="ecd60cff-708d-4a89-827e-e01025163b5a" Sep 09 20:52:27.485958 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:27.485278 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:27.485958 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:27.485417 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbvk6" podUID="0495bf90-ab82-4771-b946-a12be5bef1af" Sep 09 20:52:28.041302 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:28.041011 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd60cff-708d-4a89-827e-e01025163b5a-metrics-certs\") pod \"network-metrics-daemon-d6q8h\" (UID: \"ecd60cff-708d-4a89-827e-e01025163b5a\") " pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:28.041480 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:28.041203 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:52:28.041480 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:28.041446 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecd60cff-708d-4a89-827e-e01025163b5a-metrics-certs podName:ecd60cff-708d-4a89-827e-e01025163b5a nodeName:}" failed. No retries permitted until 2026-09-09 20:52:36.041425674 +0000 UTC m=+17.083425466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecd60cff-708d-4a89-827e-e01025163b5a-metrics-certs") pod "network-metrics-daemon-d6q8h" (UID: "ecd60cff-708d-4a89-827e-e01025163b5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:52:28.142957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:28.142309 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b2c736d-7746-4285-93bc-5ffd114396f1-original-pull-secret\") pod \"global-pull-secret-syncer-kf998\" (UID: \"0b2c736d-7746-4285-93bc-5ffd114396f1\") " pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:28.142957 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:28.142393 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk5vd\" (UniqueName: \"kubernetes.io/projected/0495bf90-ab82-4771-b946-a12be5bef1af-kube-api-access-rk5vd\") pod \"network-check-target-bbvk6\" (UID: \"0495bf90-ab82-4771-b946-a12be5bef1af\") " pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:28.142957 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:28.142524 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 20:52:28.142957 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:28.142546 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 20:52:28.142957 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:28.142562 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 20:52:28.142957 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:28.142576 2562 projected.go:196] Error preparing data for projected volume kube-api-access-rk5vd for pod openshift-network-diagnostics/network-check-target-bbvk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:52:28.142957 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:28.142610 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2c736d-7746-4285-93bc-5ffd114396f1-original-pull-secret podName:0b2c736d-7746-4285-93bc-5ffd114396f1 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:36.142588718 +0000 UTC m=+17.184588522 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b2c736d-7746-4285-93bc-5ffd114396f1-original-pull-secret") pod "global-pull-secret-syncer-kf998" (UID: "0b2c736d-7746-4285-93bc-5ffd114396f1") : object "kube-system"/"original-pull-secret" not registered Sep 09 20:52:28.142957 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:28.142631 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0495bf90-ab82-4771-b946-a12be5bef1af-kube-api-access-rk5vd podName:0495bf90-ab82-4771-b946-a12be5bef1af nodeName:}" failed. No retries permitted until 2026-09-09 20:52:36.142621343 +0000 UTC m=+17.184621136 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rk5vd" (UniqueName: "kubernetes.io/projected/0495bf90-ab82-4771-b946-a12be5bef1af-kube-api-access-rk5vd") pod "network-check-target-bbvk6" (UID: "0495bf90-ab82-4771-b946-a12be5bef1af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:52:28.485162 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:28.485130 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:28.485410 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:28.485258 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6q8h" podUID="ecd60cff-708d-4a89-827e-e01025163b5a" Sep 09 20:52:28.485756 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:28.485609 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:28.485756 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:28.485719 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf998" podUID="0b2c736d-7746-4285-93bc-5ffd114396f1" Sep 09 20:52:29.488789 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:29.488754 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:29.489269 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:29.488864 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbvk6" podUID="0495bf90-ab82-4771-b946-a12be5bef1af" Sep 09 20:52:30.485079 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:30.485036 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:30.485259 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:30.485043 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:30.485259 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:30.485180 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6q8h" podUID="ecd60cff-708d-4a89-827e-e01025163b5a" Sep 09 20:52:30.485259 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:30.485215 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf998" podUID="0b2c736d-7746-4285-93bc-5ffd114396f1" Sep 09 20:52:31.484642 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:31.484592 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:31.485149 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:31.484721 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbvk6" podUID="0495bf90-ab82-4771-b946-a12be5bef1af" Sep 09 20:52:32.485233 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:32.485193 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:32.485233 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:32.485196 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:32.485717 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:32.485399 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf998" podUID="0b2c736d-7746-4285-93bc-5ffd114396f1" Sep 09 20:52:32.485717 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:32.485514 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6q8h" podUID="ecd60cff-708d-4a89-827e-e01025163b5a" Sep 09 20:52:33.485293 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:33.485254 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:33.485786 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:33.485390 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbvk6" podUID="0495bf90-ab82-4771-b946-a12be5bef1af" Sep 09 20:52:34.485113 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:34.485079 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:34.485280 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:34.485087 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:34.485280 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:34.485195 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf998" podUID="0b2c736d-7746-4285-93bc-5ffd114396f1" Sep 09 20:52:34.485633 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:34.485309 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6q8h" podUID="ecd60cff-708d-4a89-827e-e01025163b5a" Sep 09 20:52:35.485249 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:35.485213 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:35.485437 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:35.485342 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbvk6" podUID="0495bf90-ab82-4771-b946-a12be5bef1af" Sep 09 20:52:36.101185 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:36.101150 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd60cff-708d-4a89-827e-e01025163b5a-metrics-certs\") pod \"network-metrics-daemon-d6q8h\" (UID: \"ecd60cff-708d-4a89-827e-e01025163b5a\") " pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:36.101365 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:36.101274 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:52:36.101365 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:36.101342 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecd60cff-708d-4a89-827e-e01025163b5a-metrics-certs podName:ecd60cff-708d-4a89-827e-e01025163b5a nodeName:}" failed. No retries permitted until 2026-09-09 20:52:52.101321663 +0000 UTC m=+33.143321466 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecd60cff-708d-4a89-827e-e01025163b5a-metrics-certs") pod "network-metrics-daemon-d6q8h" (UID: "ecd60cff-708d-4a89-827e-e01025163b5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:52:36.201851 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:36.201812 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk5vd\" (UniqueName: \"kubernetes.io/projected/0495bf90-ab82-4771-b946-a12be5bef1af-kube-api-access-rk5vd\") pod \"network-check-target-bbvk6\" (UID: \"0495bf90-ab82-4771-b946-a12be5bef1af\") " pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:36.202054 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:36.201876 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b2c736d-7746-4285-93bc-5ffd114396f1-original-pull-secret\") pod \"global-pull-secret-syncer-kf998\" (UID: \"0b2c736d-7746-4285-93bc-5ffd114396f1\") " pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:36.202054 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:36.202017 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 20:52:36.202054 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:36.202020 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 20:52:36.202054 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:36.202047 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 20:52:36.202228 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:36.202060 2562 projected.go:196] Error preparing data for projected volume kube-api-access-rk5vd for pod openshift-network-diagnostics/network-check-target-bbvk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:52:36.202228 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:36.202083 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2c736d-7746-4285-93bc-5ffd114396f1-original-pull-secret podName:0b2c736d-7746-4285-93bc-5ffd114396f1 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:52.202065051 +0000 UTC m=+33.244064851 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b2c736d-7746-4285-93bc-5ffd114396f1-original-pull-secret") pod "global-pull-secret-syncer-kf998" (UID: "0b2c736d-7746-4285-93bc-5ffd114396f1") : object "kube-system"/"original-pull-secret" not registered Sep 09 20:52:36.202228 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:36.202129 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0495bf90-ab82-4771-b946-a12be5bef1af-kube-api-access-rk5vd podName:0495bf90-ab82-4771-b946-a12be5bef1af nodeName:}" failed. No retries permitted until 2026-09-09 20:52:52.202094256 +0000 UTC m=+33.244094057 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rk5vd" (UniqueName: "kubernetes.io/projected/0495bf90-ab82-4771-b946-a12be5bef1af-kube-api-access-rk5vd") pod "network-check-target-bbvk6" (UID: "0495bf90-ab82-4771-b946-a12be5bef1af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:52:36.485034 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:36.484947 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:36.485034 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:36.484973 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:36.485259 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:36.485087 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf998" podUID="0b2c736d-7746-4285-93bc-5ffd114396f1" Sep 09 20:52:36.485259 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:36.485215 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6q8h" podUID="ecd60cff-708d-4a89-827e-e01025163b5a" Sep 09 20:52:37.484378 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:37.484345 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:37.484955 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:37.484488 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbvk6" podUID="0495bf90-ab82-4771-b946-a12be5bef1af" Sep 09 20:52:38.484271 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:38.484233 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:38.484476 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:38.484233 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:38.484476 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:38.484378 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6q8h" podUID="ecd60cff-708d-4a89-827e-e01025163b5a" Sep 09 20:52:38.484476 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:38.484450 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf998" podUID="0b2c736d-7746-4285-93bc-5ffd114396f1" Sep 09 20:52:39.485529 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:39.485483 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:39.486169 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:39.485591 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbvk6" podUID="0495bf90-ab82-4771-b946-a12be5bef1af" Sep 09 20:52:39.597600 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:39.597567 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" event={"ID":"63346b7c1fd6eaae6b1974fddafa03ee","Type":"ContainerStarted","Data":"ec17693bfbaf459fce3c6f6319ade9ca222f6ff916f3f7f21f91f64dcb2ed515"} Sep 09 20:52:39.599453 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:39.599425 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xktgp" event={"ID":"36c1a56e-45e1-4724-811b-c32706894bda","Type":"ContainerStarted","Data":"010b8789f70a7e6a9623e91c22666fb3bda1e4570e116ff700b21f368d75bcfe"} Sep 09 20:52:39.613853 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:39.613803 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" podStartSLOduration=19.613786714 podStartE2EDuration="19.613786714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:39.613395272 +0000 UTC m=+20.655395083" watchObservedRunningTime="2026-09-09 20:52:39.613786714 +0000 UTC m=+20.655786524" Sep 09 20:52:39.630728 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:39.630686 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xktgp" podStartSLOduration=2.032520804 podStartE2EDuration="20.630672149s" totalImagesPullingTime="18.598151345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:39.630386818 +0000 UTC m=+20.672386627" watchObservedRunningTime="2026-09-09 20:52:39.630672149 +0000 UTC m=+20.672671958" Sep 09 20:52:40.485480 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:40.485145 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:40.485642 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:40.485164 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:40.485642 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:40.485583 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6q8h" podUID="ecd60cff-708d-4a89-827e-e01025163b5a" Sep 09 20:52:40.485642 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:40.485623 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf998" podUID="0b2c736d-7746-4285-93bc-5ffd114396f1" Sep 09 20:52:40.607109 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:40.607070 2562 generic.go:356] "Generic (PLEG): container finished" podID="d2f2d9c9-c216-491f-b3e9-dac3643cd6db" containerID="0c6910a121459b43f1e75e43479859cc860a26a787cf5de1d29da676da4febe8" exitCode=0 Sep 09 20:52:40.607265 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:40.607163 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nptv6" event={"ID":"d2f2d9c9-c216-491f-b3e9-dac3643cd6db","Type":"ContainerDied","Data":"0c6910a121459b43f1e75e43479859cc860a26a787cf5de1d29da676da4febe8"} Sep 09 20:52:40.609112 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:40.609086 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" event={"ID":"1e6c7fc2-6de9-465f-aa30-47bfa17c6b04","Type":"ContainerStarted","Data":"d14dd6c213cf9de754ad96a8b8cf03fe70639f793df226c538ac3a3e19c6e4ab"} Sep 09 20:52:40.610881 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:40.610776 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5cx59" event={"ID":"4e4565ab-4f66-4753-b0d0-7d088a90d3da","Type":"ContainerStarted","Data":"34679cee25d0fb6b02fb014b1fb3467e72ba8bf590502ad2ed94ddbf7d11748c"} Sep 09 20:52:40.614187 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:40.614169 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crjwd_d55a4588-7960-4998-a99e-fca8f803b053/ovn-acl-logging/0.log" Sep 09 20:52:40.614568 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:40.614544 2562 generic.go:356] "Generic (PLEG): container finished" podID="d55a4588-7960-4998-a99e-fca8f803b053" containerID="b9a6ac137dbdd38aa8f42685cc5c224241efd1c9d2df316ecb944f35f8f59471" exitCode=1 Sep 09 20:52:40.614643 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:40.614610 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" event={"ID":"d55a4588-7960-4998-a99e-fca8f803b053","Type":"ContainerStarted","Data":"0e2ebcddbac6eedae4d5738f06e0adfa8523626f821adfc75bb2728c34583224"} Sep 09 20:52:40.614700 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:40.614678 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" event={"ID":"d55a4588-7960-4998-a99e-fca8f803b053","Type":"ContainerStarted","Data":"442a1598dc71283ff83b88801bc0a208e86ec211c4fafe0b13938a560fb50c85"} Sep 09 20:52:40.614760 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:40.614724 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" event={"ID":"d55a4588-7960-4998-a99e-fca8f803b053","Type":"ContainerStarted","Data":"67a43cecfbc54613c22fa7e4c19cb989235b938f6d3336f192b828f80c19f79a"} Sep 09 20:52:40.614812 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:40.614758 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" event={"ID":"d55a4588-7960-4998-a99e-fca8f803b053","Type":"ContainerStarted","Data":"3c9acfa202bc66af3b88bea44ee4763bf31ff986afb9a9cd13907eb167fa24d3"} Sep 09 20:52:40.614812 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:40.614806 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" event={"ID":"d55a4588-7960-4998-a99e-fca8f803b053","Type":"ContainerStarted","Data":"9c4adf5a11486475589b5ffef4a117c395e7028f252899ee5f541a8454a172b2"} Sep 09 20:52:40.614905 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:40.614854 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" event={"ID":"d55a4588-7960-4998-a99e-fca8f803b053","Type":"ContainerDied","Data":"b9a6ac137dbdd38aa8f42685cc5c224241efd1c9d2df316ecb944f35f8f59471"} Sep 09 20:52:40.616423 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:40.616400 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n7kn4" event={"ID":"7c46d28d-4f9c-4307-827a-9c9a31213cb2","Type":"ContainerStarted","Data":"fd811d9bb5edc793dd480b10a375a1700aec5c0ea41d40a59c4eb73959cd9411"} Sep 09 20:52:40.618236 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:40.618210 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-69jvv" event={"ID":"346ba7a6-b7db-48ac-9dd8-5793933dd7cd","Type":"ContainerStarted","Data":"86fc9ad460d1b186ae615e93c4883003b26c7aebf2ac247579c8f43e322e7773"} Sep 09 20:52:40.620169 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:40.620146 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gvw72" event={"ID":"a88d1a84-874b-464b-b9c7-8952100b363e","Type":"ContainerStarted","Data":"6cadd69dc261f16555da7d6f5e3906147f7fe18925d5df46571ef6e0ee8b577d"} Sep 09 20:52:40.621978 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:40.621951 2562 generic.go:356] "Generic (PLEG): container finished" podID="c3ad17323fa8ad3d6d259e0479cf3859" containerID="01d5c8b4ca9b2664a0b31ce77eca0c20b0b84adbb74b760049093972cf997dc5" exitCode=0 Sep 09 20:52:40.622068 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:40.622047 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" event={"ID":"c3ad17323fa8ad3d6d259e0479cf3859","Type":"ContainerDied","Data":"01d5c8b4ca9b2664a0b31ce77eca0c20b0b84adbb74b760049093972cf997dc5"} Sep 09 20:52:40.661413 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:40.661345 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gvw72" podStartSLOduration=3.453877111 podStartE2EDuration="21.661319425s" totalImagesPullingTime="18.207442314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:40.6612833 +0000 UTC m=+21.703283114" watchObservedRunningTime="2026-09-09 20:52:40.661319425 +0000 UTC m=+21.703319235" Sep 09 20:52:40.677868 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:40.677809 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-69jvv" podStartSLOduration=2.8754383 podStartE2EDuration="21.677787565s" totalImagesPullingTime="18.802349265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:40.677612414 +0000 UTC m=+21.719612223" watchObservedRunningTime="2026-09-09 20:52:40.677787565 +0000 UTC m=+21.719787375" Sep 09 20:52:40.707577 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:40.707533 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-n7kn4" podStartSLOduration=21.707515808 podStartE2EDuration="21.707515808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-09 20:52:40.692294259 +0000 UTC m=+21.734294068" watchObservedRunningTime="2026-09-09 20:52:40.707515808 +0000 UTC m=+21.749515616" Sep 09 20:52:40.723520 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:40.723461 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5cx59" podStartSLOduration=3.125459722 podStartE2EDuration="21.723438856s" totalImagesPullingTime="18.597979134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:40.722768355 +0000 UTC m=+21.764768165" watchObservedRunningTime="2026-09-09 20:52:40.723438856 +0000 UTC m=+21.765438667" Sep 09 20:52:40.978513 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:40.978491 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 09 20:52:41.437252 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:41.437096 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T20:52:40.978508978Z","UUID":"d2ce34e8-6a94-4f4d-a006-1e474a2f7a74","Handler":null,"Name":"","Endpoint":""} Sep 09 20:52:41.441409 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:41.438894 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 09 20:52:41.441409 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:41.438947 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 09 20:52:41.487250 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:41.487225 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:41.487788 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:41.487315 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbvk6" podUID="0495bf90-ab82-4771-b946-a12be5bef1af" Sep 09 20:52:41.626780 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:41.626741 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ftj24" event={"ID":"c30f7176-7e1b-40de-815d-8ad13d883631","Type":"ContainerStarted","Data":"5048cf819a38a622419f4c14e1e59977d724a3319a1db45a7ab02e8e9f5efd2a"} Sep 09 20:52:41.629039 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:41.628968 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" event={"ID":"c3ad17323fa8ad3d6d259e0479cf3859","Type":"ContainerStarted","Data":"74d017837549b69befd22ba0253674b4eaba31288eee1673160f0e4d75634dc1"} Sep 09 20:52:41.631284 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:41.631242 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" event={"ID":"1e6c7fc2-6de9-465f-aa30-47bfa17c6b04","Type":"ContainerStarted","Data":"9723f4c72ce8cb04fc943ee10870fcaf62782bae7ad00c2b0880113f563a91b9"} Sep 09 20:52:41.642629 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:41.642581 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ftj24" podStartSLOduration=4.011995973 podStartE2EDuration="22.642566828s" totalImagesPullingTime="18.630570855s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:41.642090927 +0000 UTC m=+22.684090735" watchObservedRunningTime="2026-09-09 20:52:41.642566828 +0000 UTC m=+22.684566701" Sep 09 20:52:41.657520 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:41.657462 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" podStartSLOduration=21.657442643 podStartE2EDuration="21.657442643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:41.656713056 +0000 UTC m=+22.698712865" watchObservedRunningTime="2026-09-09 20:52:41.657442643 +0000 UTC m=+22.699442455" Sep 09 20:52:42.415847 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:42.415798 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:42.485192 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:42.485103 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:42.485353 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:42.485103 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:42.485353 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:42.485215 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf998" podUID="0b2c736d-7746-4285-93bc-5ffd114396f1" Sep 09 20:52:42.485353 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:42.485277 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6q8h" podUID="ecd60cff-708d-4a89-827e-e01025163b5a" Sep 09 20:52:42.509522 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:42.509489 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:42.636568 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:42.636532 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" event={"ID":"1e6c7fc2-6de9-465f-aa30-47bfa17c6b04","Type":"ContainerStarted","Data":"caafa3faa49cd6f71a460aae64ea45a9496c170409b340d0c7fd1f0c0f76f825"} Sep 09 20:52:42.642319 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:42.642298 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crjwd_d55a4588-7960-4998-a99e-fca8f803b053/ovn-acl-logging/0.log" Sep 09 20:52:42.643150 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:42.642817 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" event={"ID":"d55a4588-7960-4998-a99e-fca8f803b053","Type":"ContainerStarted","Data":"c4cc946f1b3e46767e35741792a68e1c5f08ab9f1af30d19b043165c03e3108b"} Sep 09 20:52:42.657946 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:42.657866 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gsvpk" podStartSLOduration=3.006674337 podStartE2EDuration="23.657850633s" totalImagesPullingTime="20.651176296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:42.657593319 +0000 UTC m=+23.699593143" watchObservedRunningTime="2026-09-09 20:52:42.657850633 +0000 UTC m=+23.699850441" Sep 09 20:52:43.485177 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:43.484951 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:43.485365 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:43.485254 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbvk6" podUID="0495bf90-ab82-4771-b946-a12be5bef1af" Sep 09 20:52:44.013504 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:44.013463 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gvw72" Sep 09 20:52:44.014072 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:44.014050 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gvw72" Sep 09 20:52:44.484651 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:44.484618 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:44.484651 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:44.484645 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:44.484947 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:44.484740 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6q8h" podUID="ecd60cff-708d-4a89-827e-e01025163b5a" Sep 09 20:52:44.484947 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:44.484867 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf998" podUID="0b2c736d-7746-4285-93bc-5ffd114396f1" Sep 09 20:52:44.648509 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:44.648395 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gvw72" Sep 09 20:52:44.648946 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:44.648910 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gvw72" Sep 09 20:52:44.733468 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:44.733435 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:45.484397 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:45.484349 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:45.484891 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:45.484512 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbvk6" podUID="0495bf90-ab82-4771-b946-a12be5bef1af" Sep 09 20:52:45.511576 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:45.511543 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:45.610760 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:45.610723 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:45.709377 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:45.709322 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:45.811441 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:45.811365 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:46.151762 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:46.151729 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:46.253028 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:46.252843 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:46.484328 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:46.484294 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:46.484500 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:46.484338 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:46.484500 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:46.484395 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf998" podUID="0b2c736d-7746-4285-93bc-5ffd114396f1" Sep 09 20:52:46.484500 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:46.484443 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6q8h" podUID="ecd60cff-708d-4a89-827e-e01025163b5a" Sep 09 20:52:46.549308 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:46.549279 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:46.650582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:46.650473 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:46.653824 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:46.653793 2562 generic.go:356] "Generic (PLEG): container finished" podID="d2f2d9c9-c216-491f-b3e9-dac3643cd6db" containerID="07a93efcbca5de8d190c87ae83c26ad4403b49e666de1a57131c3972ac75d23f" exitCode=0 Sep 09 20:52:46.653973 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:46.653872 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nptv6" event={"ID":"d2f2d9c9-c216-491f-b3e9-dac3643cd6db","Type":"ContainerDied","Data":"07a93efcbca5de8d190c87ae83c26ad4403b49e666de1a57131c3972ac75d23f"} Sep 09 20:52:46.657422 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:46.657405 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crjwd_d55a4588-7960-4998-a99e-fca8f803b053/ovn-acl-logging/0.log" Sep 09 20:52:46.657822 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:46.657802 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" event={"ID":"d55a4588-7960-4998-a99e-fca8f803b053","Type":"ContainerStarted","Data":"2e36bc2fa737d8abd503153817db7518c6220370d6ff979c06461312d12d6681"} Sep 09 20:52:46.658300 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:46.658285 2562 scope.go:117] "RemoveContainer" containerID="b9a6ac137dbdd38aa8f42685cc5c224241efd1c9d2df316ecb944f35f8f59471" Sep 09 20:52:46.879256 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:46.879219 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:46.945187 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:46.945093 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:47.360192 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:47.360152 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:47.437232 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:47.437195 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:47.484576 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:47.484543 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:47.485221 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:47.484673 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbvk6" podUID="0495bf90-ab82-4771-b946-a12be5bef1af" Sep 09 20:52:47.665484 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:47.665410 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crjwd_d55a4588-7960-4998-a99e-fca8f803b053/ovn-acl-logging/0.log" Sep 09 20:52:47.665856 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:47.665825 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" event={"ID":"d55a4588-7960-4998-a99e-fca8f803b053","Type":"ContainerStarted","Data":"0068923a3836c597cbdc179fc3e503ece12dc9d6d9ade25cc25bf27f2aa87a92"} Sep 09 20:52:47.666143 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:47.666118 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:47.666272 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:47.666204 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:47.666272 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:47.666253 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:47.683978 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:47.683945 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:47.684121 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:47.684099 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:52:47.701034 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:47.700981 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" podStartSLOduration=10.033841046 podStartE2EDuration="28.700961801s" totalImagesPullingTime="18.667120755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:47.700021831 +0000 UTC m=+28.742021640" watchObservedRunningTime="2026-09-09 20:52:47.700961801 +0000 UTC m=+28.742961611" Sep 09 20:52:47.935579 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:47.935422 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kf998"] Sep 09 20:52:47.935579 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:47.935597 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:47.935834 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:47.935725 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf998" podUID="0b2c736d-7746-4285-93bc-5ffd114396f1" Sep 09 20:52:47.939866 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:47.939811 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bbvk6"] Sep 09 20:52:47.940056 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:47.939974 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:47.940125 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:47.940064 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbvk6" podUID="0495bf90-ab82-4771-b946-a12be5bef1af" Sep 09 20:52:47.940510 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:47.940488 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d6q8h"] Sep 09 20:52:47.940602 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:47.940589 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:47.940699 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:47.940676 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6q8h" podUID="ecd60cff-708d-4a89-827e-e01025163b5a" Sep 09 20:52:48.670527 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:48.670496 2562 generic.go:356] "Generic (PLEG): container finished" podID="d2f2d9c9-c216-491f-b3e9-dac3643cd6db" containerID="56ce59f7213babc3204b0770f0ce2491e039366bf4b0e3f892634939dda80283" exitCode=0 Sep 09 20:52:48.671108 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:48.670589 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nptv6" event={"ID":"d2f2d9c9-c216-491f-b3e9-dac3643cd6db","Type":"ContainerDied","Data":"56ce59f7213babc3204b0770f0ce2491e039366bf4b0e3f892634939dda80283"} Sep 09 20:52:49.485505 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:49.485325 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:49.485673 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:49.485402 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:49.485673 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:49.485597 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6q8h" podUID="ecd60cff-708d-4a89-827e-e01025163b5a" Sep 09 20:52:49.485673 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:49.485426 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:49.485832 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:49.485683 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbvk6" podUID="0495bf90-ab82-4771-b946-a12be5bef1af" Sep 09 20:52:49.485832 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:49.485771 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf998" podUID="0b2c736d-7746-4285-93bc-5ffd114396f1" Sep 09 20:52:50.678283 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:50.678248 2562 generic.go:356] "Generic (PLEG): container finished" podID="d2f2d9c9-c216-491f-b3e9-dac3643cd6db" containerID="a3b972201464999b445d68f9dcce7114e82e739d84c82b25217ea6080ffdd9b3" exitCode=0 Sep 09 20:52:50.678675 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:50.678296 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nptv6" event={"ID":"d2f2d9c9-c216-491f-b3e9-dac3643cd6db","Type":"ContainerDied","Data":"a3b972201464999b445d68f9dcce7114e82e739d84c82b25217ea6080ffdd9b3"} Sep 09 20:52:51.485112 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:51.484724 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:51.485112 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:51.484739 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:51.485112 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:51.484839 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf998" podUID="0b2c736d-7746-4285-93bc-5ffd114396f1" Sep 09 20:52:51.485112 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:51.484860 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:51.485112 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:51.484962 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6q8h" podUID="ecd60cff-708d-4a89-827e-e01025163b5a" Sep 09 20:52:51.485112 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:51.485048 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbvk6" podUID="0495bf90-ab82-4771-b946-a12be5bef1af" Sep 09 20:52:52.112567 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:52.112530 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd60cff-708d-4a89-827e-e01025163b5a-metrics-certs\") pod \"network-metrics-daemon-d6q8h\" (UID: \"ecd60cff-708d-4a89-827e-e01025163b5a\") " pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:52.113033 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:52.112753 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:52:52.113033 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:52.112842 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecd60cff-708d-4a89-827e-e01025163b5a-metrics-certs podName:ecd60cff-708d-4a89-827e-e01025163b5a nodeName:}" failed. No retries permitted until 2026-09-09 20:53:24.11282164 +0000 UTC m=+65.154821444 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecd60cff-708d-4a89-827e-e01025163b5a-metrics-certs") pod "network-metrics-daemon-d6q8h" (UID: "ecd60cff-708d-4a89-827e-e01025163b5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 20:52:52.213414 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:52.213373 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk5vd\" (UniqueName: \"kubernetes.io/projected/0495bf90-ab82-4771-b946-a12be5bef1af-kube-api-access-rk5vd\") pod \"network-check-target-bbvk6\" (UID: \"0495bf90-ab82-4771-b946-a12be5bef1af\") " pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:52.213588 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:52.213428 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b2c736d-7746-4285-93bc-5ffd114396f1-original-pull-secret\") pod \"global-pull-secret-syncer-kf998\" (UID: \"0b2c736d-7746-4285-93bc-5ffd114396f1\") " pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:52.213588 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:52.213539 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 20:52:52.213588 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:52.213547 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 20:52:52.213588 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:52.213568 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 20:52:52.213588 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:52.213579 2562 projected.go:196] Error preparing data for projected volume kube-api-access-rk5vd for pod openshift-network-diagnostics/network-check-target-bbvk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:52:52.213744 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:52.213591 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2c736d-7746-4285-93bc-5ffd114396f1-original-pull-secret podName:0b2c736d-7746-4285-93bc-5ffd114396f1 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:24.213579542 +0000 UTC m=+65.255579328 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b2c736d-7746-4285-93bc-5ffd114396f1-original-pull-secret") pod "global-pull-secret-syncer-kf998" (UID: "0b2c736d-7746-4285-93bc-5ffd114396f1") : object "kube-system"/"original-pull-secret" not registered Sep 09 20:52:52.213744 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:52.213618 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0495bf90-ab82-4771-b946-a12be5bef1af-kube-api-access-rk5vd podName:0495bf90-ab82-4771-b946-a12be5bef1af nodeName:}" failed. No retries permitted until 2026-09-09 20:53:24.213607702 +0000 UTC m=+65.255607489 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rk5vd" (UniqueName: "kubernetes.io/projected/0495bf90-ab82-4771-b946-a12be5bef1af-kube-api-access-rk5vd") pod "network-check-target-bbvk6" (UID: "0495bf90-ab82-4771-b946-a12be5bef1af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 20:52:53.485043 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.484946 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:53.485043 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.484968 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:53.485043 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.484977 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:53.485633 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:53.485055 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kf998" podUID="0b2c736d-7746-4285-93bc-5ffd114396f1" Sep 09 20:52:53.485633 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:53.485117 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6q8h" podUID="ecd60cff-708d-4a89-827e-e01025163b5a" Sep 09 20:52:53.485633 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:53.485176 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbvk6" podUID="0495bf90-ab82-4771-b946-a12be5bef1af" Sep 09 20:52:53.657521 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.657491 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeReady" Sep 09 20:52:53.657724 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.657635 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 09 20:52:53.708256 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.708215 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hrmkf"] Sep 09 20:52:53.732380 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.732345 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-c5n7q"] Sep 09 20:52:53.732564 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.732438 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hrmkf" Sep 09 20:52:53.734811 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.734785 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 09 20:52:53.735003 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.734835 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 09 20:52:53.735003 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.734849 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7vdcc\"" Sep 09 20:52:53.746597 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.746571 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hrmkf"] Sep 09 20:52:53.746731 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.746616 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c5n7q"] Sep 09 20:52:53.746731 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.746698 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c5n7q" Sep 09 20:52:53.749744 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.749555 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 09 20:52:53.749888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.749784 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-f6qjq\"" Sep 09 20:52:53.749888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.749833 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 09 20:52:53.750052 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.750000 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 09 20:52:53.825005 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.824971 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/92eda8aa-9402-4df8-b58e-5468b16a888e-config-volume\") pod \"dns-default-hrmkf\" (UID: \"92eda8aa-9402-4df8-b58e-5468b16a888e\") " pod="openshift-dns/dns-default-hrmkf" Sep 09 20:52:53.825239 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.825029 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92eda8aa-9402-4df8-b58e-5468b16a888e-tmp-dir\") pod \"dns-default-hrmkf\" (UID: \"92eda8aa-9402-4df8-b58e-5468b16a888e\") " pod="openshift-dns/dns-default-hrmkf" Sep 09 20:52:53.825239 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.825115 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd6d00ed-c027-4de3-962d-edd04fbf3bd5-cert\") pod \"ingress-canary-c5n7q\" (UID: \"bd6d00ed-c027-4de3-962d-edd04fbf3bd5\") " pod="openshift-ingress-canary/ingress-canary-c5n7q" Sep 09 20:52:53.825239 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.825150 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfh2q\" (UniqueName: \"kubernetes.io/projected/92eda8aa-9402-4df8-b58e-5468b16a888e-kube-api-access-lfh2q\") pod \"dns-default-hrmkf\" (UID: \"92eda8aa-9402-4df8-b58e-5468b16a888e\") " pod="openshift-dns/dns-default-hrmkf" Sep 09 20:52:53.825239 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.825219 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvwdv\" (UniqueName: \"kubernetes.io/projected/bd6d00ed-c027-4de3-962d-edd04fbf3bd5-kube-api-access-vvwdv\") pod \"ingress-canary-c5n7q\" (UID: \"bd6d00ed-c027-4de3-962d-edd04fbf3bd5\") " pod="openshift-ingress-canary/ingress-canary-c5n7q" Sep 09 20:52:53.825427 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.825298 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92eda8aa-9402-4df8-b58e-5468b16a888e-metrics-tls\") pod \"dns-default-hrmkf\" (UID: \"92eda8aa-9402-4df8-b58e-5468b16a888e\") " pod="openshift-dns/dns-default-hrmkf" Sep 09 20:52:53.858696 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.858660 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:53.926405 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.926370 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92eda8aa-9402-4df8-b58e-5468b16a888e-tmp-dir\") pod \"dns-default-hrmkf\" (UID: \"92eda8aa-9402-4df8-b58e-5468b16a888e\") " pod="openshift-dns/dns-default-hrmkf" Sep 09 20:52:53.926405 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.926415 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd6d00ed-c027-4de3-962d-edd04fbf3bd5-cert\") pod \"ingress-canary-c5n7q\" (UID: \"bd6d00ed-c027-4de3-962d-edd04fbf3bd5\") " pod="openshift-ingress-canary/ingress-canary-c5n7q" Sep 09 20:52:53.926700 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.926438 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfh2q\" (UniqueName: \"kubernetes.io/projected/92eda8aa-9402-4df8-b58e-5468b16a888e-kube-api-access-lfh2q\") pod \"dns-default-hrmkf\" (UID: \"92eda8aa-9402-4df8-b58e-5468b16a888e\") " pod="openshift-dns/dns-default-hrmkf" Sep 09 20:52:53.926700 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.926496 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvwdv\" (UniqueName: \"kubernetes.io/projected/bd6d00ed-c027-4de3-962d-edd04fbf3bd5-kube-api-access-vvwdv\") pod \"ingress-canary-c5n7q\" (UID: \"bd6d00ed-c027-4de3-962d-edd04fbf3bd5\") " pod="openshift-ingress-canary/ingress-canary-c5n7q" Sep 09 20:52:53.926700 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.926558 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92eda8aa-9402-4df8-b58e-5468b16a888e-metrics-tls\") pod \"dns-default-hrmkf\" (UID: \"92eda8aa-9402-4df8-b58e-5468b16a888e\") " pod="openshift-dns/dns-default-hrmkf" Sep 09 20:52:53.926700 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.926574 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/92eda8aa-9402-4df8-b58e-5468b16a888e-config-volume\") pod \"dns-default-hrmkf\" (UID: \"92eda8aa-9402-4df8-b58e-5468b16a888e\") " pod="openshift-dns/dns-default-hrmkf" Sep 09 20:52:53.926901 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:53.926835 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 20:52:53.926985 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:53.926904 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92eda8aa-9402-4df8-b58e-5468b16a888e-metrics-tls podName:92eda8aa-9402-4df8-b58e-5468b16a888e nodeName:}" failed. No retries permitted until 2026-09-09 20:52:54.426880752 +0000 UTC m=+35.468880542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/92eda8aa-9402-4df8-b58e-5468b16a888e-metrics-tls") pod "dns-default-hrmkf" (UID: "92eda8aa-9402-4df8-b58e-5468b16a888e") : secret "dns-default-metrics-tls" not found Sep 09 20:52:53.927175 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.927157 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/92eda8aa-9402-4df8-b58e-5468b16a888e-config-volume\") pod \"dns-default-hrmkf\" (UID: \"92eda8aa-9402-4df8-b58e-5468b16a888e\") " pod="openshift-dns/dns-default-hrmkf" Sep 09 20:52:53.927275 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.927195 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92eda8aa-9402-4df8-b58e-5468b16a888e-tmp-dir\") pod \"dns-default-hrmkf\" (UID: \"92eda8aa-9402-4df8-b58e-5468b16a888e\") " pod="openshift-dns/dns-default-hrmkf" Sep 09 20:52:53.931909 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.931728 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd6d00ed-c027-4de3-962d-edd04fbf3bd5-cert\") pod \"ingress-canary-c5n7q\" (UID: \"bd6d00ed-c027-4de3-962d-edd04fbf3bd5\") " pod="openshift-ingress-canary/ingress-canary-c5n7q" Sep 09 20:52:53.936964 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.936907 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfh2q\" (UniqueName: \"kubernetes.io/projected/92eda8aa-9402-4df8-b58e-5468b16a888e-kube-api-access-lfh2q\") pod \"dns-default-hrmkf\" (UID: \"92eda8aa-9402-4df8-b58e-5468b16a888e\") " pod="openshift-dns/dns-default-hrmkf" Sep 09 20:52:53.937501 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.937475 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvwdv\" (UniqueName: \"kubernetes.io/projected/bd6d00ed-c027-4de3-962d-edd04fbf3bd5-kube-api-access-vvwdv\") pod \"ingress-canary-c5n7q\" (UID: \"bd6d00ed-c027-4de3-962d-edd04fbf3bd5\") " pod="openshift-ingress-canary/ingress-canary-c5n7q" Sep 09 20:52:53.957533 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:53.957474 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:54.055897 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:54.055811 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c5n7q" Sep 09 20:52:54.430644 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:54.430617 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92eda8aa-9402-4df8-b58e-5468b16a888e-metrics-tls\") pod \"dns-default-hrmkf\" (UID: \"92eda8aa-9402-4df8-b58e-5468b16a888e\") " pod="openshift-dns/dns-default-hrmkf" Sep 09 20:52:54.430948 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:54.430779 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 20:52:54.430948 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:54.430838 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92eda8aa-9402-4df8-b58e-5468b16a888e-metrics-tls podName:92eda8aa-9402-4df8-b58e-5468b16a888e nodeName:}" failed. No retries permitted until 2026-09-09 20:52:55.430820448 +0000 UTC m=+36.472820236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/92eda8aa-9402-4df8-b58e-5468b16a888e-metrics-tls") pod "dns-default-hrmkf" (UID: "92eda8aa-9402-4df8-b58e-5468b16a888e") : secret "dns-default-metrics-tls" not found Sep 09 20:52:55.371816 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:55.371778 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c5n7q"] Sep 09 20:52:55.375999 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:55.375969 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6d00ed_c027_4de3_962d_edd04fbf3bd5.slice/crio-dfd4d0f926cd8a25189cbd7b7813e0b26ecdbc12871628135e5b70144efa3f33 WatchSource:0}: Error finding container dfd4d0f926cd8a25189cbd7b7813e0b26ecdbc12871628135e5b70144efa3f33: Status 404 returned error can't find the container with id dfd4d0f926cd8a25189cbd7b7813e0b26ecdbc12871628135e5b70144efa3f33 Sep 09 20:52:55.439124 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:55.439098 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92eda8aa-9402-4df8-b58e-5468b16a888e-metrics-tls\") pod \"dns-default-hrmkf\" (UID: \"92eda8aa-9402-4df8-b58e-5468b16a888e\") " pod="openshift-dns/dns-default-hrmkf" Sep 09 20:52:55.439262 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:55.439240 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 20:52:55.439327 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:55.439294 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92eda8aa-9402-4df8-b58e-5468b16a888e-metrics-tls podName:92eda8aa-9402-4df8-b58e-5468b16a888e nodeName:}" failed. No retries permitted until 2026-09-09 20:52:57.439280433 +0000 UTC m=+38.481280223 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/92eda8aa-9402-4df8-b58e-5468b16a888e-metrics-tls") pod "dns-default-hrmkf" (UID: "92eda8aa-9402-4df8-b58e-5468b16a888e") : secret "dns-default-metrics-tls" not found Sep 09 20:52:55.485027 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:55.484994 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:52:55.485205 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:55.484994 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:52:55.485339 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:55.485318 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:52:55.487717 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:55.487693 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w9kf4\"" Sep 09 20:52:55.488477 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:55.488445 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 20:52:55.488477 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:55.488469 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 20:52:55.488684 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:55.488480 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 20:52:55.488684 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:55.488544 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7m4wj\"" Sep 09 20:52:55.488810 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:55.488785 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 20:52:55.692971 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:55.692882 2562 generic.go:356] "Generic (PLEG): container finished" podID="d2f2d9c9-c216-491f-b3e9-dac3643cd6db" containerID="06effc9824e64edefcc57fa20cdf0dbfb0e679f99086ec258f87c44619df3e63" exitCode=0 Sep 09 20:52:55.693129 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:55.692974 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nptv6" event={"ID":"d2f2d9c9-c216-491f-b3e9-dac3643cd6db","Type":"ContainerDied","Data":"06effc9824e64edefcc57fa20cdf0dbfb0e679f99086ec258f87c44619df3e63"} Sep 09 20:52:55.694243 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:55.694218 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c5n7q" event={"ID":"bd6d00ed-c027-4de3-962d-edd04fbf3bd5","Type":"ContainerStarted","Data":"dfd4d0f926cd8a25189cbd7b7813e0b26ecdbc12871628135e5b70144efa3f33"} Sep 09 20:52:56.700994 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:56.700897 2562 generic.go:356] "Generic (PLEG): container finished" podID="d2f2d9c9-c216-491f-b3e9-dac3643cd6db" containerID="76446f1a87f7a9226f075e8e3e68ba2b510a306881214e8725dcdd04f454ec95" exitCode=0 Sep 09 20:52:56.701604 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:56.700967 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nptv6" event={"ID":"d2f2d9c9-c216-491f-b3e9-dac3643cd6db","Type":"ContainerDied","Data":"76446f1a87f7a9226f075e8e3e68ba2b510a306881214e8725dcdd04f454ec95"} Sep 09 20:52:57.457701 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:57.457443 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92eda8aa-9402-4df8-b58e-5468b16a888e-metrics-tls\") pod \"dns-default-hrmkf\" (UID: \"92eda8aa-9402-4df8-b58e-5468b16a888e\") " pod="openshift-dns/dns-default-hrmkf" Sep 09 20:52:57.457925 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:57.457604 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 20:52:57.457925 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:57.457875 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92eda8aa-9402-4df8-b58e-5468b16a888e-metrics-tls podName:92eda8aa-9402-4df8-b58e-5468b16a888e nodeName:}" failed. No retries permitted until 2026-09-09 20:53:01.457847382 +0000 UTC m=+42.499847174 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/92eda8aa-9402-4df8-b58e-5468b16a888e-metrics-tls") pod "dns-default-hrmkf" (UID: "92eda8aa-9402-4df8-b58e-5468b16a888e") : secret "dns-default-metrics-tls" not found Sep 09 20:52:57.545689 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:57.545660 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:57.643388 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:57.643358 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:57.706900 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:57.706866 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nptv6" event={"ID":"d2f2d9c9-c216-491f-b3e9-dac3643cd6db","Type":"ContainerStarted","Data":"6435705c02259267b7e6a8e61526d15600461bd4cd6f179ddca842f323f0f3cd"} Sep 09 20:52:57.737434 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:57.737337 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nptv6" podStartSLOduration=7.555917152 podStartE2EDuration="38.73731869s" totalImagesPullingTime="31.181401538s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:57.734969954 +0000 UTC m=+38.776969760" watchObservedRunningTime="2026-09-09 20:52:57.73731869 +0000 UTC m=+38.779318493" Sep 09 20:52:57.779806 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:57.779766 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:57.869044 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:57.869005 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:57.921563 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:57.921538 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-qlzzw"] Sep 09 20:52:57.944742 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:57.944488 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-qlzzw" Sep 09 20:52:57.946277 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:57.946255 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-qlzzw"] Sep 09 20:52:57.947758 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:57.947721 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 09 20:52:57.948004 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:57.947984 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 09 20:52:57.948080 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:57.948057 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-nwfnd\"" Sep 09 20:52:57.948141 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:57.948099 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 09 20:52:57.948523 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:57.948505 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 09 20:52:57.954200 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:57.954175 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 09 20:52:57.982061 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:57.982033 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:58.006851 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.006822 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-dcncg"] Sep 09 20:52:58.038441 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.038406 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-dcncg"] Sep 09 20:52:58.038610 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.038506 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-dcncg" Sep 09 20:52:58.045269 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.045246 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-rjlgw\"" Sep 09 20:52:58.061524 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.061494 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/247631db-a55e-4fbc-9bde-f921c2bb309c-serving-cert\") pod \"console-operator-85b454874b-qlzzw\" (UID: \"247631db-a55e-4fbc-9bde-f921c2bb309c\") " pod="openshift-console-operator/console-operator-85b454874b-qlzzw" Sep 09 20:52:58.061710 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.061639 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwzw9\" (UniqueName: \"kubernetes.io/projected/247631db-a55e-4fbc-9bde-f921c2bb309c-kube-api-access-fwzw9\") pod \"console-operator-85b454874b-qlzzw\" (UID: \"247631db-a55e-4fbc-9bde-f921c2bb309c\") " pod="openshift-console-operator/console-operator-85b454874b-qlzzw" Sep 09 20:52:58.061710 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.061686 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/247631db-a55e-4fbc-9bde-f921c2bb309c-config\") pod \"console-operator-85b454874b-qlzzw\" (UID: \"247631db-a55e-4fbc-9bde-f921c2bb309c\") " pod="openshift-console-operator/console-operator-85b454874b-qlzzw" Sep 09 20:52:58.061796 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.061718 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/247631db-a55e-4fbc-9bde-f921c2bb309c-trusted-ca\") pod \"console-operator-85b454874b-qlzzw\" (UID: \"247631db-a55e-4fbc-9bde-f921c2bb309c\") " pod="openshift-console-operator/console-operator-85b454874b-qlzzw" Sep 09 20:52:58.083422 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.083381 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:58.102133 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.102097 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-cljj5"] Sep 09 20:52:58.120778 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.120745 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7f7bbb8cc6-ptg74"] Sep 09 20:52:58.120950 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.120857 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-cljj5" Sep 09 20:52:58.123418 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.123391 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 09 20:52:58.123568 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.123528 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-kwj46\"" Sep 09 20:52:58.123642 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.123535 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 09 20:52:58.145629 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.145599 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-cljj5"] Sep 09 20:52:58.145629 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.145632 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f7bbb8cc6-ptg74"] Sep 09 20:52:58.145808 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.145720 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:52:58.148286 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.148264 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 09 20:52:58.148286 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.148278 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 09 20:52:58.148514 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.148498 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 09 20:52:58.148628 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.148610 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 09 20:52:58.149100 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.149080 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 09 20:52:58.149338 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.149319 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 09 20:52:58.152187 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.152167 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qn5sb\"" Sep 09 20:52:58.162549 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.162523 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwzw9\" (UniqueName: \"kubernetes.io/projected/247631db-a55e-4fbc-9bde-f921c2bb309c-kube-api-access-fwzw9\") pod \"console-operator-85b454874b-qlzzw\" (UID: \"247631db-a55e-4fbc-9bde-f921c2bb309c\") " pod="openshift-console-operator/console-operator-85b454874b-qlzzw" Sep 09 20:52:58.162549 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.162552 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/247631db-a55e-4fbc-9bde-f921c2bb309c-config\") pod \"console-operator-85b454874b-qlzzw\" (UID: \"247631db-a55e-4fbc-9bde-f921c2bb309c\") " pod="openshift-console-operator/console-operator-85b454874b-qlzzw" Sep 09 20:52:58.162775 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.162571 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/247631db-a55e-4fbc-9bde-f921c2bb309c-trusted-ca\") pod \"console-operator-85b454874b-qlzzw\" (UID: \"247631db-a55e-4fbc-9bde-f921c2bb309c\") " pod="openshift-console-operator/console-operator-85b454874b-qlzzw" Sep 09 20:52:58.162775 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.162596 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hnst\" (UniqueName: \"kubernetes.io/projected/2f682073-bfdd-4a9a-b6fd-ff564f6253d4-kube-api-access-8hnst\") pod \"network-check-source-685456655f-dcncg\" (UID: \"2f682073-bfdd-4a9a-b6fd-ff564f6253d4\") " pod="openshift-network-diagnostics/network-check-source-685456655f-dcncg" Sep 09 20:52:58.162775 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.162658 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/247631db-a55e-4fbc-9bde-f921c2bb309c-serving-cert\") pod \"console-operator-85b454874b-qlzzw\" (UID: \"247631db-a55e-4fbc-9bde-f921c2bb309c\") " pod="openshift-console-operator/console-operator-85b454874b-qlzzw" Sep 09 20:52:58.163453 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.163430 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/247631db-a55e-4fbc-9bde-f921c2bb309c-config\") pod \"console-operator-85b454874b-qlzzw\" (UID: \"247631db-a55e-4fbc-9bde-f921c2bb309c\") " pod="openshift-console-operator/console-operator-85b454874b-qlzzw" Sep 09 20:52:58.163534 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.163456 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/247631db-a55e-4fbc-9bde-f921c2bb309c-trusted-ca\") pod \"console-operator-85b454874b-qlzzw\" (UID: \"247631db-a55e-4fbc-9bde-f921c2bb309c\") " pod="openshift-console-operator/console-operator-85b454874b-qlzzw" Sep 09 20:52:58.166140 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.166117 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/247631db-a55e-4fbc-9bde-f921c2bb309c-serving-cert\") pod \"console-operator-85b454874b-qlzzw\" (UID: \"247631db-a55e-4fbc-9bde-f921c2bb309c\") " pod="openshift-console-operator/console-operator-85b454874b-qlzzw" Sep 09 20:52:58.172353 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.172328 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwzw9\" (UniqueName: \"kubernetes.io/projected/247631db-a55e-4fbc-9bde-f921c2bb309c-kube-api-access-fwzw9\") pod \"console-operator-85b454874b-qlzzw\" (UID: \"247631db-a55e-4fbc-9bde-f921c2bb309c\") " pod="openshift-console-operator/console-operator-85b454874b-qlzzw" Sep 09 20:52:58.254867 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.254828 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-qlzzw" Sep 09 20:52:58.263173 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.263090 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8d11438c-9ef0-491b-9813-36b419613f92-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cljj5\" (UID: \"8d11438c-9ef0-491b-9813-36b419613f92\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cljj5" Sep 09 20:52:58.263173 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.263133 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8d11438c-9ef0-491b-9813-36b419613f92-nginx-conf\") pod \"networking-console-plugin-79977f9844-cljj5\" (UID: \"8d11438c-9ef0-491b-9813-36b419613f92\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cljj5" Sep 09 20:52:58.263173 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.263155 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhnnq\" (UniqueName: \"kubernetes.io/projected/95fc21f9-62ed-41e4-9609-28ffab20ab40-kube-api-access-jhnnq\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:52:58.263361 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.263212 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95fc21f9-62ed-41e4-9609-28ffab20ab40-service-ca-bundle\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:52:58.263361 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.263244 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/95fc21f9-62ed-41e4-9609-28ffab20ab40-stats-auth\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:52:58.263361 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.263327 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95fc21f9-62ed-41e4-9609-28ffab20ab40-metrics-certs\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:52:58.263450 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.263370 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hnst\" (UniqueName: \"kubernetes.io/projected/2f682073-bfdd-4a9a-b6fd-ff564f6253d4-kube-api-access-8hnst\") pod \"network-check-source-685456655f-dcncg\" (UID: \"2f682073-bfdd-4a9a-b6fd-ff564f6253d4\") " pod="openshift-network-diagnostics/network-check-source-685456655f-dcncg" Sep 09 20:52:58.263450 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.263388 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/95fc21f9-62ed-41e4-9609-28ffab20ab40-default-certificate\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:52:58.266509 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.266484 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:58.275666 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.275641 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hnst\" (UniqueName: \"kubernetes.io/projected/2f682073-bfdd-4a9a-b6fd-ff564f6253d4-kube-api-access-8hnst\") pod \"network-check-source-685456655f-dcncg\" (UID: \"2f682073-bfdd-4a9a-b6fd-ff564f6253d4\") " pod="openshift-network-diagnostics/network-check-source-685456655f-dcncg" Sep 09 20:52:58.349349 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.349317 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-dcncg" Sep 09 20:52:58.362769 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.362736 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:58.363660 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.363641 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/95fc21f9-62ed-41e4-9609-28ffab20ab40-default-certificate\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:52:58.363713 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.363676 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8d11438c-9ef0-491b-9813-36b419613f92-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cljj5\" (UID: \"8d11438c-9ef0-491b-9813-36b419613f92\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cljj5" Sep 09 20:52:58.363713 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.363703 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8d11438c-9ef0-491b-9813-36b419613f92-nginx-conf\") pod \"networking-console-plugin-79977f9844-cljj5\" (UID: \"8d11438c-9ef0-491b-9813-36b419613f92\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cljj5" Sep 09 20:52:58.363794 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.363720 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhnnq\" (UniqueName: \"kubernetes.io/projected/95fc21f9-62ed-41e4-9609-28ffab20ab40-kube-api-access-jhnnq\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:52:58.363794 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.363745 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95fc21f9-62ed-41e4-9609-28ffab20ab40-service-ca-bundle\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:52:58.363794 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.363769 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/95fc21f9-62ed-41e4-9609-28ffab20ab40-stats-auth\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:52:58.363952 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:58.363795 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 20:52:58.363952 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.363831 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95fc21f9-62ed-41e4-9609-28ffab20ab40-metrics-certs\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:52:58.363952 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:58.363850 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d11438c-9ef0-491b-9813-36b419613f92-networking-console-plugin-cert podName:8d11438c-9ef0-491b-9813-36b419613f92 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:58.863834743 +0000 UTC m=+39.905834533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8d11438c-9ef0-491b-9813-36b419613f92-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-cljj5" (UID: "8d11438c-9ef0-491b-9813-36b419613f92") : secret "networking-console-plugin-cert" not found Sep 09 20:52:58.363952 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:58.363906 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 09 20:52:58.364111 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:58.363953 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95fc21f9-62ed-41e4-9609-28ffab20ab40-service-ca-bundle podName:95fc21f9-62ed-41e4-9609-28ffab20ab40 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:58.863919521 +0000 UTC m=+39.905919323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/95fc21f9-62ed-41e4-9609-28ffab20ab40-service-ca-bundle") pod "router-default-7f7bbb8cc6-ptg74" (UID: "95fc21f9-62ed-41e4-9609-28ffab20ab40") : configmap references non-existent config key: service-ca.crt Sep 09 20:52:58.364111 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:58.363986 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95fc21f9-62ed-41e4-9609-28ffab20ab40-metrics-certs podName:95fc21f9-62ed-41e4-9609-28ffab20ab40 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:58.863973245 +0000 UTC m=+39.905973037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95fc21f9-62ed-41e4-9609-28ffab20ab40-metrics-certs") pod "router-default-7f7bbb8cc6-ptg74" (UID: "95fc21f9-62ed-41e4-9609-28ffab20ab40") : secret "router-metrics-certs-default" not found Sep 09 20:52:58.364555 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.364533 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8d11438c-9ef0-491b-9813-36b419613f92-nginx-conf\") pod \"networking-console-plugin-79977f9844-cljj5\" (UID: \"8d11438c-9ef0-491b-9813-36b419613f92\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cljj5" Sep 09 20:52:58.366143 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.366120 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/95fc21f9-62ed-41e4-9609-28ffab20ab40-stats-auth\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:52:58.366224 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.366166 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/95fc21f9-62ed-41e4-9609-28ffab20ab40-default-certificate\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:52:58.375955 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.375913 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhnnq\" (UniqueName: \"kubernetes.io/projected/95fc21f9-62ed-41e4-9609-28ffab20ab40-kube-api-access-jhnnq\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:52:58.547493 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.547412 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-qlzzw"] Sep 09 20:52:58.550796 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:58.550765 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247631db_a55e_4fbc_9bde_f921c2bb309c.slice/crio-b1a90797491d10de987d2399d1f5c0a194fd1417f40009a6372808392a883c97 WatchSource:0}: Error finding container b1a90797491d10de987d2399d1f5c0a194fd1417f40009a6372808392a883c97: Status 404 returned error can't find the container with id b1a90797491d10de987d2399d1f5c0a194fd1417f40009a6372808392a883c97 Sep 09 20:52:58.685472 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:52:58.685444 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f682073_bfdd_4a9a_b6fd_ff564f6253d4.slice/crio-cf35ead1ad73e8d8010da377e9a8ef2d094c2509584ca6d8eb3405c3970d8769 WatchSource:0}: Error finding container cf35ead1ad73e8d8010da377e9a8ef2d094c2509584ca6d8eb3405c3970d8769: Status 404 returned error can't find the container with id cf35ead1ad73e8d8010da377e9a8ef2d094c2509584ca6d8eb3405c3970d8769 Sep 09 20:52:58.685841 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.685821 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-dcncg"] Sep 09 20:52:58.710597 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.710561 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c5n7q" event={"ID":"bd6d00ed-c027-4de3-962d-edd04fbf3bd5","Type":"ContainerStarted","Data":"e783f15eb9daac7de05d01d55a73dd3df009d6edb2aba492cebbc85587a9e867"} Sep 09 20:52:58.711876 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.711845 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-dcncg" event={"ID":"2f682073-bfdd-4a9a-b6fd-ff564f6253d4","Type":"ContainerStarted","Data":"cf35ead1ad73e8d8010da377e9a8ef2d094c2509584ca6d8eb3405c3970d8769"} Sep 09 20:52:58.713062 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.713039 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-qlzzw" event={"ID":"247631db-a55e-4fbc-9bde-f921c2bb309c","Type":"ContainerStarted","Data":"b1a90797491d10de987d2399d1f5c0a194fd1417f40009a6372808392a883c97"} Sep 09 20:52:58.758317 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.758256 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-c5n7q" podStartSLOduration=3.218652755 podStartE2EDuration="5.758240916s" totalImagesPullingTime="2.539588161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:58.757193126 +0000 UTC m=+39.799192936" watchObservedRunningTime="2026-09-09 20:52:58.758240916 +0000 UTC m=+39.800240722" Sep 09 20:52:58.868057 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.868012 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8d11438c-9ef0-491b-9813-36b419613f92-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cljj5\" (UID: \"8d11438c-9ef0-491b-9813-36b419613f92\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cljj5" Sep 09 20:52:58.868219 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.868095 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95fc21f9-62ed-41e4-9609-28ffab20ab40-service-ca-bundle\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:52:58.868219 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:58.868166 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 20:52:58.868343 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:58.868237 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d11438c-9ef0-491b-9813-36b419613f92-networking-console-plugin-cert podName:8d11438c-9ef0-491b-9813-36b419613f92 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:59.86821776 +0000 UTC m=+40.910217552 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8d11438c-9ef0-491b-9813-36b419613f92-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-cljj5" (UID: "8d11438c-9ef0-491b-9813-36b419613f92") : secret "networking-console-plugin-cert" not found Sep 09 20:52:58.868343 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.868310 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95fc21f9-62ed-41e4-9609-28ffab20ab40-metrics-certs\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:52:58.868463 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:58.868361 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95fc21f9-62ed-41e4-9609-28ffab20ab40-service-ca-bundle podName:95fc21f9-62ed-41e4-9609-28ffab20ab40 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:59.868343138 +0000 UTC m=+40.910342939 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/95fc21f9-62ed-41e4-9609-28ffab20ab40-service-ca-bundle") pod "router-default-7f7bbb8cc6-ptg74" (UID: "95fc21f9-62ed-41e4-9609-28ffab20ab40") : configmap references non-existent config key: service-ca.crt Sep 09 20:52:58.868463 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:58.868383 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 09 20:52:58.868463 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:58.868417 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95fc21f9-62ed-41e4-9609-28ffab20ab40-metrics-certs podName:95fc21f9-62ed-41e4-9609-28ffab20ab40 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:59.868405519 +0000 UTC m=+40.910405310 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95fc21f9-62ed-41e4-9609-28ffab20ab40-metrics-certs") pod "router-default-7f7bbb8cc6-ptg74" (UID: "95fc21f9-62ed-41e4-9609-28ffab20ab40") : secret "router-metrics-certs-default" not found Sep 09 20:52:58.980676 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:58.980635 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:59.279329 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:59.279249 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:59.410807 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:59.410769 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:52:59.876093 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:59.876048 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95fc21f9-62ed-41e4-9609-28ffab20ab40-service-ca-bundle\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:52:59.876567 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:59.876173 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95fc21f9-62ed-41e4-9609-28ffab20ab40-metrics-certs\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:52:59.876567 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:52:59.876225 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8d11438c-9ef0-491b-9813-36b419613f92-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cljj5\" (UID: \"8d11438c-9ef0-491b-9813-36b419613f92\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cljj5" Sep 09 20:52:59.876567 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:59.876295 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95fc21f9-62ed-41e4-9609-28ffab20ab40-service-ca-bundle podName:95fc21f9-62ed-41e4-9609-28ffab20ab40 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:01.87626959 +0000 UTC m=+42.918269380 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/95fc21f9-62ed-41e4-9609-28ffab20ab40-service-ca-bundle") pod "router-default-7f7bbb8cc6-ptg74" (UID: "95fc21f9-62ed-41e4-9609-28ffab20ab40") : configmap references non-existent config key: service-ca.crt Sep 09 20:52:59.876567 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:59.876345 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 20:52:59.876567 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:59.876396 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d11438c-9ef0-491b-9813-36b419613f92-networking-console-plugin-cert podName:8d11438c-9ef0-491b-9813-36b419613f92 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:01.876379884 +0000 UTC m=+42.918379677 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8d11438c-9ef0-491b-9813-36b419613f92-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-cljj5" (UID: "8d11438c-9ef0-491b-9813-36b419613f92") : secret "networking-console-plugin-cert" not found Sep 09 20:52:59.876567 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:59.876460 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 09 20:52:59.876567 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:52:59.876489 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95fc21f9-62ed-41e4-9609-28ffab20ab40-metrics-certs podName:95fc21f9-62ed-41e4-9609-28ffab20ab40 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:01.876477813 +0000 UTC m=+42.918477599 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95fc21f9-62ed-41e4-9609-28ffab20ab40-metrics-certs") pod "router-default-7f7bbb8cc6-ptg74" (UID: "95fc21f9-62ed-41e4-9609-28ffab20ab40") : secret "router-metrics-certs-default" not found Sep 09 20:53:00.641723 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:00.641670 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:00.708110 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:00.708044 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:01.490805 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:01.490772 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92eda8aa-9402-4df8-b58e-5468b16a888e-metrics-tls\") pod \"dns-default-hrmkf\" (UID: \"92eda8aa-9402-4df8-b58e-5468b16a888e\") " pod="openshift-dns/dns-default-hrmkf" Sep 09 20:53:01.491254 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:01.490965 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 20:53:01.491254 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:01.491039 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92eda8aa-9402-4df8-b58e-5468b16a888e-metrics-tls podName:92eda8aa-9402-4df8-b58e-5468b16a888e nodeName:}" failed. No retries permitted until 2026-09-09 20:53:09.491018007 +0000 UTC m=+50.533017799 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/92eda8aa-9402-4df8-b58e-5468b16a888e-metrics-tls") pod "dns-default-hrmkf" (UID: "92eda8aa-9402-4df8-b58e-5468b16a888e") : secret "dns-default-metrics-tls" not found Sep 09 20:53:01.652555 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:01.652511 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:01.842066 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:01.842029 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:01.893182 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:01.893130 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8d11438c-9ef0-491b-9813-36b419613f92-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cljj5\" (UID: \"8d11438c-9ef0-491b-9813-36b419613f92\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cljj5" Sep 09 20:53:01.893359 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:01.893212 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95fc21f9-62ed-41e4-9609-28ffab20ab40-service-ca-bundle\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:53:01.893359 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:01.893272 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95fc21f9-62ed-41e4-9609-28ffab20ab40-metrics-certs\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:53:01.893359 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:01.893288 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 20:53:01.893485 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:01.893360 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d11438c-9ef0-491b-9813-36b419613f92-networking-console-plugin-cert podName:8d11438c-9ef0-491b-9813-36b419613f92 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:05.893340946 +0000 UTC m=+46.935340744 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8d11438c-9ef0-491b-9813-36b419613f92-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-cljj5" (UID: "8d11438c-9ef0-491b-9813-36b419613f92") : secret "networking-console-plugin-cert" not found Sep 09 20:53:01.893485 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:01.893371 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 09 20:53:01.893485 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:01.893378 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95fc21f9-62ed-41e4-9609-28ffab20ab40-service-ca-bundle podName:95fc21f9-62ed-41e4-9609-28ffab20ab40 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:05.89337012 +0000 UTC m=+46.935369906 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/95fc21f9-62ed-41e4-9609-28ffab20ab40-service-ca-bundle") pod "router-default-7f7bbb8cc6-ptg74" (UID: "95fc21f9-62ed-41e4-9609-28ffab20ab40") : configmap references non-existent config key: service-ca.crt Sep 09 20:53:01.893485 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:01.893410 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95fc21f9-62ed-41e4-9609-28ffab20ab40-metrics-certs podName:95fc21f9-62ed-41e4-9609-28ffab20ab40 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:05.893399198 +0000 UTC m=+46.935398985 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95fc21f9-62ed-41e4-9609-28ffab20ab40-metrics-certs") pod "router-default-7f7bbb8cc6-ptg74" (UID: "95fc21f9-62ed-41e4-9609-28ffab20ab40") : secret "router-metrics-certs-default" not found Sep 09 20:53:02.422883 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:02.422850 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:02.523944 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:02.523902 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:02.717379 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:02.717330 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:02.724883 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:02.724848 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-dcncg" event={"ID":"2f682073-bfdd-4a9a-b6fd-ff564f6253d4","Type":"ContainerStarted","Data":"36d783092d8c323950456a302bfff4e01bfe18c0c71213dc8b30c3cf2f2d4c0e"} Sep 09 20:53:02.726539 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:02.726512 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-qlzzw" event={"ID":"247631db-a55e-4fbc-9bde-f921c2bb309c","Type":"ContainerStarted","Data":"da1fb68b8b0b72060046b13c746f576894e59ca20fdfea868a245449d7e59f8f"} Sep 09 20:53:02.726766 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:02.726724 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-qlzzw" Sep 09 20:53:02.728413 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:02.728379 2562 patch_prober.go:28] interesting pod/console-operator-85b454874b-qlzzw container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.133.0.8:8443/readyz\": dial tcp 10.133.0.8:8443: connect: connection refused" start-of-body= Sep 09 20:53:02.728504 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:02.728432 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-85b454874b-qlzzw" podUID="247631db-a55e-4fbc-9bde-f921c2bb309c" containerName="console-operator" probeResult="failure" output="Get \"https://10.133.0.8:8443/readyz\": dial tcp 10.133.0.8:8443: connect: connection refused" Sep 09 20:53:02.742240 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:02.742186 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-dcncg" podStartSLOduration=1.870437435 podStartE2EDuration="5.742167804s" totalImagesPullingTime="3.871730369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:02.741713691 +0000 UTC m=+43.783713512" watchObservedRunningTime="2026-09-09 20:53:02.742167804 +0000 UTC m=+43.784167617" Sep 09 20:53:02.761708 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:02.761653 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-qlzzw" podStartSLOduration=1.7646345270000001 podStartE2EDuration="5.761636457s" totalImagesPullingTime="3.99700193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:02.761005965 +0000 UTC m=+43.803005774" watchObservedRunningTime="2026-09-09 20:53:02.761636457 +0000 UTC m=+43.803636266" Sep 09 20:53:02.819399 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:02.819358 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:03.198224 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.198184 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:03.272803 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.272773 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kfg2r"] Sep 09 20:53:03.297559 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.297516 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:03.298862 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.298843 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kfg2r"] Sep 09 20:53:03.299000 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.298985 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:03.301513 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.301476 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 09 20:53:03.301513 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.301494 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 09 20:53:03.301675 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.301496 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s99rg\"" Sep 09 20:53:03.301675 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.301487 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 09 20:53:03.301675 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.301496 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 09 20:53:03.406580 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.406539 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d9680a3c-cfee-45a8-8793-8a5d5888935b-crio-socket\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:03.406580 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.406582 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9680a3c-cfee-45a8-8793-8a5d5888935b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:03.406783 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.406604 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9680a3c-cfee-45a8-8793-8a5d5888935b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:03.406783 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.406731 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qwgx\" (UniqueName: \"kubernetes.io/projected/d9680a3c-cfee-45a8-8793-8a5d5888935b-kube-api-access-9qwgx\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:03.406924 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.406905 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9680a3c-cfee-45a8-8793-8a5d5888935b-data-volume\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:03.507452 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.507348 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9680a3c-cfee-45a8-8793-8a5d5888935b-data-volume\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:03.507452 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.507396 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d9680a3c-cfee-45a8-8793-8a5d5888935b-crio-socket\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:03.507452 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.507423 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9680a3c-cfee-45a8-8793-8a5d5888935b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:03.507452 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.507445 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9680a3c-cfee-45a8-8793-8a5d5888935b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:03.507786 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.507509 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qwgx\" (UniqueName: \"kubernetes.io/projected/d9680a3c-cfee-45a8-8793-8a5d5888935b-kube-api-access-9qwgx\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:03.507786 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:03.507523 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 20:53:03.507786 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:03.507595 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9680a3c-cfee-45a8-8793-8a5d5888935b-kube-rbac-proxy-cm podName:d9680a3c-cfee-45a8-8793-8a5d5888935b nodeName:}" failed. No retries permitted until 2026-09-09 20:53:04.007576937 +0000 UTC m=+45.049576724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d9680a3c-cfee-45a8-8793-8a5d5888935b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kfg2r" (UID: "d9680a3c-cfee-45a8-8793-8a5d5888935b") : configmap "kube-rbac-proxy" not found Sep 09 20:53:03.507786 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.507613 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d9680a3c-cfee-45a8-8793-8a5d5888935b-crio-socket\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:03.507786 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:03.507621 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 20:53:03.507786 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:03.507685 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9680a3c-cfee-45a8-8793-8a5d5888935b-insights-runtime-extractor-tls podName:d9680a3c-cfee-45a8-8793-8a5d5888935b nodeName:}" failed. No retries permitted until 2026-09-09 20:53:04.007671287 +0000 UTC m=+45.049671073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d9680a3c-cfee-45a8-8793-8a5d5888935b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kfg2r" (UID: "d9680a3c-cfee-45a8-8793-8a5d5888935b") : secret "insights-runtime-extractor-tls" not found Sep 09 20:53:03.507786 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.507770 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9680a3c-cfee-45a8-8793-8a5d5888935b-data-volume\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:03.523959 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.523906 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qwgx\" (UniqueName: \"kubernetes.io/projected/d9680a3c-cfee-45a8-8793-8a5d5888935b-kube-api-access-9qwgx\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:03.730886 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.730856 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-qlzzw_247631db-a55e-4fbc-9bde-f921c2bb309c/console-operator/0.log" Sep 09 20:53:03.731084 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.730895 2562 generic.go:356] "Generic (PLEG): container finished" podID="247631db-a55e-4fbc-9bde-f921c2bb309c" containerID="da1fb68b8b0b72060046b13c746f576894e59ca20fdfea868a245449d7e59f8f" exitCode=255 Sep 09 20:53:03.731084 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.730947 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-qlzzw" event={"ID":"247631db-a55e-4fbc-9bde-f921c2bb309c","Type":"ContainerDied","Data":"da1fb68b8b0b72060046b13c746f576894e59ca20fdfea868a245449d7e59f8f"} Sep 09 20:53:03.731195 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.731179 2562 scope.go:117] "RemoveContainer" containerID="da1fb68b8b0b72060046b13c746f576894e59ca20fdfea868a245449d7e59f8f" Sep 09 20:53:03.767964 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:03.767843 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:04.012796 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:04.012754 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9680a3c-cfee-45a8-8793-8a5d5888935b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:04.012796 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:04.012794 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9680a3c-cfee-45a8-8793-8a5d5888935b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:04.013096 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:04.012897 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 20:53:04.013096 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:04.012990 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9680a3c-cfee-45a8-8793-8a5d5888935b-kube-rbac-proxy-cm podName:d9680a3c-cfee-45a8-8793-8a5d5888935b nodeName:}" failed. No retries permitted until 2026-09-09 20:53:05.012968284 +0000 UTC m=+46.054968077 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d9680a3c-cfee-45a8-8793-8a5d5888935b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kfg2r" (UID: "d9680a3c-cfee-45a8-8793-8a5d5888935b") : configmap "kube-rbac-proxy" not found Sep 09 20:53:04.013096 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:04.013037 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 20:53:04.013096 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:04.013085 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9680a3c-cfee-45a8-8793-8a5d5888935b-insights-runtime-extractor-tls podName:d9680a3c-cfee-45a8-8793-8a5d5888935b nodeName:}" failed. No retries permitted until 2026-09-09 20:53:05.0130719 +0000 UTC m=+46.055071688 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d9680a3c-cfee-45a8-8793-8a5d5888935b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kfg2r" (UID: "d9680a3c-cfee-45a8-8793-8a5d5888935b") : secret "insights-runtime-extractor-tls" not found Sep 09 20:53:04.363411 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:04.363368 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:04.478943 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:04.478895 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:04.734480 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:04.734422 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-qlzzw_247631db-a55e-4fbc-9bde-f921c2bb309c/console-operator/1.log" Sep 09 20:53:04.734847 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:04.734832 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-qlzzw_247631db-a55e-4fbc-9bde-f921c2bb309c/console-operator/0.log" Sep 09 20:53:04.734893 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:04.734867 2562 generic.go:356] "Generic (PLEG): container finished" podID="247631db-a55e-4fbc-9bde-f921c2bb309c" containerID="935250c5ba41b4e86663b4e5366f3a0b5f3ca4f784c49e880d6cdec0c4661e58" exitCode=255 Sep 09 20:53:04.734954 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:04.734904 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-qlzzw" event={"ID":"247631db-a55e-4fbc-9bde-f921c2bb309c","Type":"ContainerDied","Data":"935250c5ba41b4e86663b4e5366f3a0b5f3ca4f784c49e880d6cdec0c4661e58"} Sep 09 20:53:04.735006 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:04.734960 2562 scope.go:117] "RemoveContainer" containerID="da1fb68b8b0b72060046b13c746f576894e59ca20fdfea868a245449d7e59f8f" Sep 09 20:53:04.735171 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:04.735157 2562 scope.go:117] "RemoveContainer" containerID="935250c5ba41b4e86663b4e5366f3a0b5f3ca4f784c49e880d6cdec0c4661e58" Sep 09 20:53:04.735337 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:04.735319 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-qlzzw_openshift-console-operator(247631db-a55e-4fbc-9bde-f921c2bb309c)\"" pod="openshift-console-operator/console-operator-85b454874b-qlzzw" podUID="247631db-a55e-4fbc-9bde-f921c2bb309c" Sep 09 20:53:04.815692 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:04.815658 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-tkd5g"] Sep 09 20:53:04.846183 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:04.846152 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-tkd5g"] Sep 09 20:53:04.846341 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:04.846271 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-tkd5g" Sep 09 20:53:04.869561 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:04.869531 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 09 20:53:04.869561 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:04.869537 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 09 20:53:04.869726 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:04.869582 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 09 20:53:04.877826 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:04.877803 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 09 20:53:04.878229 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:04.878215 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5pftw\"" Sep 09 20:53:05.022883 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:05.022798 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9680a3c-cfee-45a8-8793-8a5d5888935b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:05.022883 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:05.022835 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9680a3c-cfee-45a8-8793-8a5d5888935b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:05.022883 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:05.022859 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfh46\" (UniqueName: \"kubernetes.io/projected/7ea22503-a60a-4c81-b80a-01dabdfb1aab-kube-api-access-lfh46\") pod \"service-ca-75d99c484-tkd5g\" (UID: \"7ea22503-a60a-4c81-b80a-01dabdfb1aab\") " pod="openshift-service-ca/service-ca-75d99c484-tkd5g" Sep 09 20:53:05.023147 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:05.022951 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 20:53:05.023147 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:05.022956 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7ea22503-a60a-4c81-b80a-01dabdfb1aab-signing-cabundle\") pod \"service-ca-75d99c484-tkd5g\" (UID: \"7ea22503-a60a-4c81-b80a-01dabdfb1aab\") " pod="openshift-service-ca/service-ca-75d99c484-tkd5g" Sep 09 20:53:05.023147 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:05.023013 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9680a3c-cfee-45a8-8793-8a5d5888935b-kube-rbac-proxy-cm podName:d9680a3c-cfee-45a8-8793-8a5d5888935b nodeName:}" failed. No retries permitted until 2026-09-09 20:53:07.022995623 +0000 UTC m=+48.064995409 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d9680a3c-cfee-45a8-8793-8a5d5888935b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kfg2r" (UID: "d9680a3c-cfee-45a8-8793-8a5d5888935b") : configmap "kube-rbac-proxy" not found Sep 09 20:53:05.023147 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:05.023021 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 20:53:05.023147 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:05.023074 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9680a3c-cfee-45a8-8793-8a5d5888935b-insights-runtime-extractor-tls podName:d9680a3c-cfee-45a8-8793-8a5d5888935b nodeName:}" failed. No retries permitted until 2026-09-09 20:53:07.023058569 +0000 UTC m=+48.065058359 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d9680a3c-cfee-45a8-8793-8a5d5888935b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kfg2r" (UID: "d9680a3c-cfee-45a8-8793-8a5d5888935b") : secret "insights-runtime-extractor-tls" not found Sep 09 20:53:05.023147 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:05.023113 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7ea22503-a60a-4c81-b80a-01dabdfb1aab-signing-key\") pod \"service-ca-75d99c484-tkd5g\" (UID: \"7ea22503-a60a-4c81-b80a-01dabdfb1aab\") " pod="openshift-service-ca/service-ca-75d99c484-tkd5g" Sep 09 20:53:05.047194 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:05.047163 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:05.124320 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:05.124279 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfh46\" (UniqueName: \"kubernetes.io/projected/7ea22503-a60a-4c81-b80a-01dabdfb1aab-kube-api-access-lfh46\") pod \"service-ca-75d99c484-tkd5g\" (UID: \"7ea22503-a60a-4c81-b80a-01dabdfb1aab\") " pod="openshift-service-ca/service-ca-75d99c484-tkd5g" Sep 09 20:53:05.124492 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:05.124348 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7ea22503-a60a-4c81-b80a-01dabdfb1aab-signing-cabundle\") pod \"service-ca-75d99c484-tkd5g\" (UID: \"7ea22503-a60a-4c81-b80a-01dabdfb1aab\") " pod="openshift-service-ca/service-ca-75d99c484-tkd5g" Sep 09 20:53:05.124492 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:05.124405 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7ea22503-a60a-4c81-b80a-01dabdfb1aab-signing-key\") pod \"service-ca-75d99c484-tkd5g\" (UID: \"7ea22503-a60a-4c81-b80a-01dabdfb1aab\") " pod="openshift-service-ca/service-ca-75d99c484-tkd5g" Sep 09 20:53:05.125040 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:05.124984 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7ea22503-a60a-4c81-b80a-01dabdfb1aab-signing-cabundle\") pod \"service-ca-75d99c484-tkd5g\" (UID: \"7ea22503-a60a-4c81-b80a-01dabdfb1aab\") " pod="openshift-service-ca/service-ca-75d99c484-tkd5g" Sep 09 20:53:05.126826 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:05.126801 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7ea22503-a60a-4c81-b80a-01dabdfb1aab-signing-key\") pod \"service-ca-75d99c484-tkd5g\" (UID: \"7ea22503-a60a-4c81-b80a-01dabdfb1aab\") " pod="openshift-service-ca/service-ca-75d99c484-tkd5g" Sep 09 20:53:05.137665 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:05.137631 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfh46\" (UniqueName: \"kubernetes.io/projected/7ea22503-a60a-4c81-b80a-01dabdfb1aab-kube-api-access-lfh46\") pod \"service-ca-75d99c484-tkd5g\" (UID: \"7ea22503-a60a-4c81-b80a-01dabdfb1aab\") " pod="openshift-service-ca/service-ca-75d99c484-tkd5g" Sep 09 20:53:05.147623 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:05.147593 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:05.154464 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:05.154438 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-tkd5g" Sep 09 20:53:05.426197 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:05.426172 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n7kn4_7c46d28d-4f9c-4307-827a-9c9a31213cb2/dns-node-resolver/0.log" Sep 09 20:53:05.438363 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:05.438323 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-tkd5g"] Sep 09 20:53:05.441756 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:53:05.441727 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea22503_a60a_4c81_b80a_01dabdfb1aab.slice/crio-707f995ec5dd97461c9d849d205e6676b5a028b7b46a78b8012e7f81dc2701e9 WatchSource:0}: Error finding container 707f995ec5dd97461c9d849d205e6676b5a028b7b46a78b8012e7f81dc2701e9: Status 404 returned error can't find the container with id 707f995ec5dd97461c9d849d205e6676b5a028b7b46a78b8012e7f81dc2701e9 Sep 09 20:53:05.739367 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:05.739284 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-qlzzw_247631db-a55e-4fbc-9bde-f921c2bb309c/console-operator/1.log" Sep 09 20:53:05.739958 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:05.739661 2562 scope.go:117] "RemoveContainer" containerID="935250c5ba41b4e86663b4e5366f3a0b5f3ca4f784c49e880d6cdec0c4661e58" Sep 09 20:53:05.739958 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:05.739810 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-qlzzw_openshift-console-operator(247631db-a55e-4fbc-9bde-f921c2bb309c)\"" pod="openshift-console-operator/console-operator-85b454874b-qlzzw" podUID="247631db-a55e-4fbc-9bde-f921c2bb309c" Sep 09 20:53:05.740752 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:05.740726 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-tkd5g" event={"ID":"7ea22503-a60a-4c81-b80a-01dabdfb1aab","Type":"ContainerStarted","Data":"707f995ec5dd97461c9d849d205e6676b5a028b7b46a78b8012e7f81dc2701e9"} Sep 09 20:53:05.930913 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:05.930878 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95fc21f9-62ed-41e4-9609-28ffab20ab40-service-ca-bundle\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:53:05.931115 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:05.931069 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95fc21f9-62ed-41e4-9609-28ffab20ab40-service-ca-bundle podName:95fc21f9-62ed-41e4-9609-28ffab20ab40 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:13.931047923 +0000 UTC m=+54.973047714 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/95fc21f9-62ed-41e4-9609-28ffab20ab40-service-ca-bundle") pod "router-default-7f7bbb8cc6-ptg74" (UID: "95fc21f9-62ed-41e4-9609-28ffab20ab40") : configmap references non-existent config key: service-ca.crt Sep 09 20:53:05.931190 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:05.931116 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95fc21f9-62ed-41e4-9609-28ffab20ab40-metrics-certs\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:53:05.931242 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:05.931192 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 09 20:53:05.931242 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:05.931222 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95fc21f9-62ed-41e4-9609-28ffab20ab40-metrics-certs podName:95fc21f9-62ed-41e4-9609-28ffab20ab40 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:13.931212044 +0000 UTC m=+54.973211830 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95fc21f9-62ed-41e4-9609-28ffab20ab40-metrics-certs") pod "router-default-7f7bbb8cc6-ptg74" (UID: "95fc21f9-62ed-41e4-9609-28ffab20ab40") : secret "router-metrics-certs-default" not found Sep 09 20:53:05.931354 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:05.931279 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8d11438c-9ef0-491b-9813-36b419613f92-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cljj5\" (UID: \"8d11438c-9ef0-491b-9813-36b419613f92\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cljj5" Sep 09 20:53:05.931412 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:05.931388 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 20:53:05.931467 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:05.931426 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d11438c-9ef0-491b-9813-36b419613f92-networking-console-plugin-cert podName:8d11438c-9ef0-491b-9813-36b419613f92 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:13.931417735 +0000 UTC m=+54.973417522 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8d11438c-9ef0-491b-9813-36b419613f92-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-cljj5" (UID: "8d11438c-9ef0-491b-9813-36b419613f92") : secret "networking-console-plugin-cert" not found Sep 09 20:53:05.981471 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:05.981432 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:06.026125 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:06.026054 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5cx59_4e4565ab-4f66-4753-b0d0-7d088a90d3da/node-ca/0.log" Sep 09 20:53:06.825224 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:06.825187 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c5n7q_bd6d00ed-c027-4de3-962d-edd04fbf3bd5/serve-healthcheck-canary/0.log" Sep 09 20:53:07.041206 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:07.041166 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9680a3c-cfee-45a8-8793-8a5d5888935b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:07.041206 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:07.041204 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9680a3c-cfee-45a8-8793-8a5d5888935b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:07.041448 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:07.041304 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 20:53:07.041448 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:07.041388 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9680a3c-cfee-45a8-8793-8a5d5888935b-kube-rbac-proxy-cm podName:d9680a3c-cfee-45a8-8793-8a5d5888935b nodeName:}" failed. No retries permitted until 2026-09-09 20:53:11.041369547 +0000 UTC m=+52.083369334 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d9680a3c-cfee-45a8-8793-8a5d5888935b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kfg2r" (UID: "d9680a3c-cfee-45a8-8793-8a5d5888935b") : configmap "kube-rbac-proxy" not found Sep 09 20:53:07.041448 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:07.041408 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 20:53:07.041577 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:07.041474 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9680a3c-cfee-45a8-8793-8a5d5888935b-insights-runtime-extractor-tls podName:d9680a3c-cfee-45a8-8793-8a5d5888935b nodeName:}" failed. No retries permitted until 2026-09-09 20:53:11.041456363 +0000 UTC m=+52.083456157 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d9680a3c-cfee-45a8-8793-8a5d5888935b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kfg2r" (UID: "d9680a3c-cfee-45a8-8793-8a5d5888935b") : secret "insights-runtime-extractor-tls" not found Sep 09 20:53:08.255589 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:08.255545 2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-qlzzw" Sep 09 20:53:08.256016 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:08.255913 2562 scope.go:117] "RemoveContainer" containerID="935250c5ba41b4e86663b4e5366f3a0b5f3ca4f784c49e880d6cdec0c4661e58" Sep 09 20:53:08.256102 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:08.256083 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-qlzzw_openshift-console-operator(247631db-a55e-4fbc-9bde-f921c2bb309c)\"" pod="openshift-console-operator/console-operator-85b454874b-qlzzw" podUID="247631db-a55e-4fbc-9bde-f921c2bb309c" Sep 09 20:53:08.750264 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:08.750228 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-tkd5g" event={"ID":"7ea22503-a60a-4c81-b80a-01dabdfb1aab","Type":"ContainerStarted","Data":"2d6d0f400f2de121d3740402093861a9f3c5f579a4216f724d17e779ab4c1aa8"} Sep 09 20:53:08.773169 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:08.773118 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-tkd5g" podStartSLOduration=2.4860212280000002 podStartE2EDuration="4.773102624s" totalImagesPullingTime="2.287081396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:08.772959899 +0000 UTC m=+49.814959706" watchObservedRunningTime="2026-09-09 20:53:08.773102624 +0000 UTC m=+49.815102432" Sep 09 20:53:09.319717 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:09.319677 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:09.428122 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:09.428037 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:09.522070 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:09.522038 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:09.557248 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:09.557212 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92eda8aa-9402-4df8-b58e-5468b16a888e-metrics-tls\") pod \"dns-default-hrmkf\" (UID: \"92eda8aa-9402-4df8-b58e-5468b16a888e\") " pod="openshift-dns/dns-default-hrmkf" Sep 09 20:53:09.557763 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:09.557571 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 20:53:09.557866 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:09.557835 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92eda8aa-9402-4df8-b58e-5468b16a888e-metrics-tls podName:92eda8aa-9402-4df8-b58e-5468b16a888e nodeName:}" failed. No retries permitted until 2026-09-09 20:53:25.557814939 +0000 UTC m=+66.599814738 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/92eda8aa-9402-4df8-b58e-5468b16a888e-metrics-tls") pod "dns-default-hrmkf" (UID: "92eda8aa-9402-4df8-b58e-5468b16a888e") : secret "dns-default-metrics-tls" not found Sep 09 20:53:09.622629 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:09.622596 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:10.294452 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:10.294415 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:10.397806 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:10.397773 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:10.612320 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:10.612286 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:10.818146 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:10.818112 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:11.008384 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:11.008299 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:11.069867 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:11.069814 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9680a3c-cfee-45a8-8793-8a5d5888935b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:11.069867 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:11.069873 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9680a3c-cfee-45a8-8793-8a5d5888935b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:11.070119 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:11.069954 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 20:53:11.070119 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:11.070017 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9680a3c-cfee-45a8-8793-8a5d5888935b-kube-rbac-proxy-cm podName:d9680a3c-cfee-45a8-8793-8a5d5888935b nodeName:}" failed. No retries permitted until 2026-09-09 20:53:19.070002253 +0000 UTC m=+60.112002041 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d9680a3c-cfee-45a8-8793-8a5d5888935b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kfg2r" (UID: "d9680a3c-cfee-45a8-8793-8a5d5888935b") : configmap "kube-rbac-proxy" not found Sep 09 20:53:11.070119 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:11.070047 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 20:53:11.070239 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:11.070125 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9680a3c-cfee-45a8-8793-8a5d5888935b-insights-runtime-extractor-tls podName:d9680a3c-cfee-45a8-8793-8a5d5888935b nodeName:}" failed. No retries permitted until 2026-09-09 20:53:19.070106241 +0000 UTC m=+60.112106028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d9680a3c-cfee-45a8-8793-8a5d5888935b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kfg2r" (UID: "d9680a3c-cfee-45a8-8793-8a5d5888935b") : secret "insights-runtime-extractor-tls" not found Sep 09 20:53:11.414767 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:11.414726 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:12.025282 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:12.025234 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:12.617478 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:12.617442 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:12.727082 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:12.727030 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-qlzzw" Sep 09 20:53:12.727423 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:12.727410 2562 scope.go:117] "RemoveContainer" containerID="935250c5ba41b4e86663b4e5366f3a0b5f3ca4f784c49e880d6cdec0c4661e58" Sep 09 20:53:12.727574 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:12.727557 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-qlzzw_openshift-console-operator(247631db-a55e-4fbc-9bde-f921c2bb309c)\"" pod="openshift-console-operator/console-operator-85b454874b-qlzzw" podUID="247631db-a55e-4fbc-9bde-f921c2bb309c" Sep 09 20:53:12.815270 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:12.815236 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:13.017405 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:13.017312 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:13.091261 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:13.091223 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:13.339947 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:13.339898 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:13.441127 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:13.441093 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:13.996046 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:13.996006 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8d11438c-9ef0-491b-9813-36b419613f92-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cljj5\" (UID: \"8d11438c-9ef0-491b-9813-36b419613f92\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cljj5" Sep 09 20:53:13.996433 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:13.996099 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95fc21f9-62ed-41e4-9609-28ffab20ab40-service-ca-bundle\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:53:13.996433 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:13.996166 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 20:53:13.996433 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:13.996183 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95fc21f9-62ed-41e4-9609-28ffab20ab40-metrics-certs\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:53:13.996433 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:13.996239 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d11438c-9ef0-491b-9813-36b419613f92-networking-console-plugin-cert podName:8d11438c-9ef0-491b-9813-36b419613f92 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:29.996215176 +0000 UTC m=+71.038214976 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8d11438c-9ef0-491b-9813-36b419613f92-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-cljj5" (UID: "8d11438c-9ef0-491b-9813-36b419613f92") : secret "networking-console-plugin-cert" not found Sep 09 20:53:13.996433 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:13.996259 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95fc21f9-62ed-41e4-9609-28ffab20ab40-service-ca-bundle podName:95fc21f9-62ed-41e4-9609-28ffab20ab40 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:29.996249563 +0000 UTC m=+71.038249351 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/95fc21f9-62ed-41e4-9609-28ffab20ab40-service-ca-bundle") pod "router-default-7f7bbb8cc6-ptg74" (UID: "95fc21f9-62ed-41e4-9609-28ffab20ab40") : configmap references non-existent config key: service-ca.crt Sep 09 20:53:13.998986 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:13.998963 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95fc21f9-62ed-41e4-9609-28ffab20ab40-metrics-certs\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:53:14.216543 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:14.216511 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:14.280903 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:14.280813 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:14.622634 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:14.622591 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:14.815767 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:14.815734 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:15.005383 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:15.005295 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:15.208096 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:15.208059 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:15.924032 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:15.923991 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:16.150099 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:16.150060 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:16.251458 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:16.251367 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:16.316877 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:16.316835 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:16.593260 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:16.593231 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:16.652996 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:16.652957 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:16.806491 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:16.806433 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:17.022368 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:17.022283 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:17.097145 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:17.097111 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:17.408989 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:17.408956 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:18.018943 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:18.018894 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:18.432650 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:18.432604 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:18.834521 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:18.834428 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:19.136788 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:19.136747 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9680a3c-cfee-45a8-8793-8a5d5888935b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:19.136788 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:19.136788 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9680a3c-cfee-45a8-8793-8a5d5888935b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:19.137038 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:19.136952 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 20:53:19.137038 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:19.137007 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9680a3c-cfee-45a8-8793-8a5d5888935b-insights-runtime-extractor-tls podName:d9680a3c-cfee-45a8-8793-8a5d5888935b nodeName:}" failed. No retries permitted until 2026-09-09 20:53:35.136993451 +0000 UTC m=+76.178993237 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d9680a3c-cfee-45a8-8793-8a5d5888935b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kfg2r" (UID: "d9680a3c-cfee-45a8-8793-8a5d5888935b") : secret "insights-runtime-extractor-tls" not found Sep 09 20:53:19.137331 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:19.137311 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9680a3c-cfee-45a8-8793-8a5d5888935b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:19.308298 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:19.308264 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:19.435495 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:19.435414 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:19.689530 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:19.689452 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-crjwd" Sep 09 20:53:19.835662 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:19.835579 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:20.096613 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:20.096583 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:24.172590 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:24.172538 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd60cff-708d-4a89-827e-e01025163b5a-metrics-certs\") pod \"network-metrics-daemon-d6q8h\" (UID: \"ecd60cff-708d-4a89-827e-e01025163b5a\") " pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:53:24.175188 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:24.175169 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 20:53:24.185158 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:24.185129 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd60cff-708d-4a89-827e-e01025163b5a-metrics-certs\") pod \"network-metrics-daemon-d6q8h\" (UID: \"ecd60cff-708d-4a89-827e-e01025163b5a\") " pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:53:24.272889 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:24.272846 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk5vd\" (UniqueName: \"kubernetes.io/projected/0495bf90-ab82-4771-b946-a12be5bef1af-kube-api-access-rk5vd\") pod \"network-check-target-bbvk6\" (UID: \"0495bf90-ab82-4771-b946-a12be5bef1af\") " pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:53:24.273075 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:24.272948 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b2c736d-7746-4285-93bc-5ffd114396f1-original-pull-secret\") pod \"global-pull-secret-syncer-kf998\" (UID: \"0b2c736d-7746-4285-93bc-5ffd114396f1\") " pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:53:24.275437 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:24.275408 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk5vd\" (UniqueName: \"kubernetes.io/projected/0495bf90-ab82-4771-b946-a12be5bef1af-kube-api-access-rk5vd\") pod \"network-check-target-bbvk6\" (UID: \"0495bf90-ab82-4771-b946-a12be5bef1af\") " pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:53:24.275677 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:24.275664 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 20:53:24.285161 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:24.285131 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b2c736d-7746-4285-93bc-5ffd114396f1-original-pull-secret\") pod \"global-pull-secret-syncer-kf998\" (UID: \"0b2c736d-7746-4285-93bc-5ffd114396f1\") " pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:53:24.300775 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:24.300749 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7m4wj\"" Sep 09 20:53:24.307374 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:24.307350 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w9kf4\"" Sep 09 20:53:24.308341 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:24.308326 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:53:24.309221 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:24.309205 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kf998" Sep 09 20:53:24.315258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:24.315238 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6q8h" Sep 09 20:53:24.601951 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:24.601821 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bbvk6"] Sep 09 20:53:24.604406 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:24.604328 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d6q8h"] Sep 09 20:53:24.605094 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:53:24.605073 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0495bf90_ab82_4771_b946_a12be5bef1af.slice/crio-123358b43568a49659a4725fb0dc17440e02e01794e637cb07a7f7ea6ade7db8 WatchSource:0}: Error finding container 123358b43568a49659a4725fb0dc17440e02e01794e637cb07a7f7ea6ade7db8: Status 404 returned error can't find the container with id 123358b43568a49659a4725fb0dc17440e02e01794e637cb07a7f7ea6ade7db8 Sep 09 20:53:24.607875 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:24.607849 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kf998"] Sep 09 20:53:24.608309 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:53:24.608281 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd60cff_708d_4a89_827e_e01025163b5a.slice/crio-fb15a086eb6e608566e342f1e38fafafb10913ce223b4cd228f811e19c576698 WatchSource:0}: Error finding container fb15a086eb6e608566e342f1e38fafafb10913ce223b4cd228f811e19c576698: Status 404 returned error can't find the container with id fb15a086eb6e608566e342f1e38fafafb10913ce223b4cd228f811e19c576698 Sep 09 20:53:24.611925 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:53:24.611904 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b2c736d_7746_4285_93bc_5ffd114396f1.slice/crio-fe5248fe63845890b3fe5d16ce63edf8cce0008af13e66e6c5654dea29ca91d9 WatchSource:0}: Error finding container fe5248fe63845890b3fe5d16ce63edf8cce0008af13e66e6c5654dea29ca91d9: Status 404 returned error can't find the container with id fe5248fe63845890b3fe5d16ce63edf8cce0008af13e66e6c5654dea29ca91d9 Sep 09 20:53:24.794364 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:24.794331 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kf998" event={"ID":"0b2c736d-7746-4285-93bc-5ffd114396f1","Type":"ContainerStarted","Data":"fe5248fe63845890b3fe5d16ce63edf8cce0008af13e66e6c5654dea29ca91d9"} Sep 09 20:53:24.795808 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:24.795787 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bbvk6" event={"ID":"0495bf90-ab82-4771-b946-a12be5bef1af","Type":"ContainerStarted","Data":"3475b4498770f0221d12c8140275301b69c99b8154ddfbdc3f8dce3ae9366fe3"} Sep 09 20:53:24.795959 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:24.795820 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bbvk6" event={"ID":"0495bf90-ab82-4771-b946-a12be5bef1af","Type":"ContainerStarted","Data":"123358b43568a49659a4725fb0dc17440e02e01794e637cb07a7f7ea6ade7db8"} Sep 09 20:53:24.795959 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:24.795874 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:53:24.796975 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:24.796952 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d6q8h" event={"ID":"ecd60cff-708d-4a89-827e-e01025163b5a","Type":"ContainerStarted","Data":"fb15a086eb6e608566e342f1e38fafafb10913ce223b4cd228f811e19c576698"} Sep 09 20:53:24.812521 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:24.812473 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bbvk6" podStartSLOduration=65.812457233 podStartE2EDuration="1m5.812457233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:24.811566893 +0000 UTC m=+65.853566703" watchObservedRunningTime="2026-09-09 20:53:24.812457233 +0000 UTC m=+65.854457042" Sep 09 20:53:25.485873 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:25.485512 2562 scope.go:117] "RemoveContainer" containerID="935250c5ba41b4e86663b4e5366f3a0b5f3ca4f784c49e880d6cdec0c4661e58" Sep 09 20:53:25.583257 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:25.583218 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92eda8aa-9402-4df8-b58e-5468b16a888e-metrics-tls\") pod \"dns-default-hrmkf\" (UID: \"92eda8aa-9402-4df8-b58e-5468b16a888e\") " pod="openshift-dns/dns-default-hrmkf" Sep 09 20:53:25.586237 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:25.586203 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92eda8aa-9402-4df8-b58e-5468b16a888e-metrics-tls\") pod \"dns-default-hrmkf\" (UID: \"92eda8aa-9402-4df8-b58e-5468b16a888e\") " pod="openshift-dns/dns-default-hrmkf" Sep 09 20:53:25.803171 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:25.803090 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-qlzzw_247631db-a55e-4fbc-9bde-f921c2bb309c/console-operator/1.log" Sep 09 20:53:25.803339 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:25.803203 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-qlzzw" event={"ID":"247631db-a55e-4fbc-9bde-f921c2bb309c","Type":"ContainerStarted","Data":"6ea92158e4068e79644e0ad001fde91185ef73157ba624ea92e59e48a9ff1470"} Sep 09 20:53:25.803731 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:25.803702 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-qlzzw" Sep 09 20:53:25.846946 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:25.846898 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7vdcc\"" Sep 09 20:53:25.854402 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:25.854370 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hrmkf" Sep 09 20:53:26.132672 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.132638 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-qlzzw" Sep 09 20:53:26.153502 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.150081 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hrmkf"] Sep 09 20:53:26.155200 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:53:26.155168 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92eda8aa_9402_4df8_b58e_5468b16a888e.slice/crio-7f381b214ddff391fb890903a302dfc79fc21e37cd1534e4a2a37dac2416cb24 WatchSource:0}: Error finding container 7f381b214ddff391fb890903a302dfc79fc21e37cd1534e4a2a37dac2416cb24: Status 404 returned error can't find the container with id 7f381b214ddff391fb890903a302dfc79fc21e37cd1534e4a2a37dac2416cb24 Sep 09 20:53:26.321690 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.321608 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:26.425751 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.425596 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:26.678711 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.678628 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69f8df4d98-hwmp7"] Sep 09 20:53:26.681892 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.681865 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69f8df4d98-hwmp7" Sep 09 20:53:26.691671 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.690804 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 09 20:53:26.692375 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.692203 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 09 20:53:26.692716 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.692581 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 09 20:53:26.694133 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.693500 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 09 20:53:26.708486 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.708413 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69f8df4d98-hwmp7"] Sep 09 20:53:26.712470 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.712381 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:26.782857 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.782814 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-jrfbb"] Sep 09 20:53:26.797428 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.797389 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1467-bcd3-44a0-b1dd-ec1b2cadf2f5-tmp\") pod \"klusterlet-addon-workmgr-69f8df4d98-hwmp7\" (UID: \"0f3e1467-bcd3-44a0-b1dd-ec1b2cadf2f5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69f8df4d98-hwmp7" Sep 09 20:53:26.797616 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.797452 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5qtl\" (UniqueName: \"kubernetes.io/projected/0f3e1467-bcd3-44a0-b1dd-ec1b2cadf2f5-kube-api-access-l5qtl\") pod \"klusterlet-addon-workmgr-69f8df4d98-hwmp7\" (UID: \"0f3e1467-bcd3-44a0-b1dd-ec1b2cadf2f5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69f8df4d98-hwmp7" Sep 09 20:53:26.797616 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.797502 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0f3e1467-bcd3-44a0-b1dd-ec1b2cadf2f5-klusterlet-config\") pod \"klusterlet-addon-workmgr-69f8df4d98-hwmp7\" (UID: \"0f3e1467-bcd3-44a0-b1dd-ec1b2cadf2f5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69f8df4d98-hwmp7" Sep 09 20:53:26.807515 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.807483 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-jrfbb"] Sep 09 20:53:26.807699 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.807603 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-jrfbb" Sep 09 20:53:26.810951 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.810904 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 09 20:53:26.811584 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.811197 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 09 20:53:26.811584 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.811477 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-f4kt4\"" Sep 09 20:53:26.811584 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.811486 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hrmkf" event={"ID":"92eda8aa-9402-4df8-b58e-5468b16a888e","Type":"ContainerStarted","Data":"7f381b214ddff391fb890903a302dfc79fc21e37cd1534e4a2a37dac2416cb24"} Sep 09 20:53:26.816203 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.816106 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d6q8h" event={"ID":"ecd60cff-708d-4a89-827e-e01025163b5a","Type":"ContainerStarted","Data":"dda1cf588190a445cf1387b439710006ce788aa4ab082dc36b3913d911b8d347"} Sep 09 20:53:26.816203 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.816171 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d6q8h" event={"ID":"ecd60cff-708d-4a89-827e-e01025163b5a","Type":"ContainerStarted","Data":"641ef0e5b1b879cda238ba0e5501a5c02fc0feafbdf95f42fd1785bf0479ee07"} Sep 09 20:53:26.860694 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.860652 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:26.874014 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.873950 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-d6q8h" podStartSLOduration=66.011368822 podStartE2EDuration="1m7.873912266s" totalImagesPullingTime="1.862543444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:26.872097156 +0000 UTC m=+67.914096965" watchObservedRunningTime="2026-09-09 20:53:26.873912266 +0000 UTC m=+67.915912075" Sep 09 20:53:26.898922 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.898745 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1467-bcd3-44a0-b1dd-ec1b2cadf2f5-tmp\") pod \"klusterlet-addon-workmgr-69f8df4d98-hwmp7\" (UID: \"0f3e1467-bcd3-44a0-b1dd-ec1b2cadf2f5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69f8df4d98-hwmp7" Sep 09 20:53:26.898922 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.898830 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvwrv\" (UniqueName: \"kubernetes.io/projected/1ee25625-54d3-4ac1-912c-42f78f271480-kube-api-access-pvwrv\") pod \"downloads-85d98c998f-jrfbb\" (UID: \"1ee25625-54d3-4ac1-912c-42f78f271480\") " pod="openshift-console/downloads-85d98c998f-jrfbb" Sep 09 20:53:26.898922 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.898863 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5qtl\" (UniqueName: \"kubernetes.io/projected/0f3e1467-bcd3-44a0-b1dd-ec1b2cadf2f5-kube-api-access-l5qtl\") pod \"klusterlet-addon-workmgr-69f8df4d98-hwmp7\" (UID: \"0f3e1467-bcd3-44a0-b1dd-ec1b2cadf2f5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69f8df4d98-hwmp7" Sep 09 20:53:26.899246 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.898922 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0f3e1467-bcd3-44a0-b1dd-ec1b2cadf2f5-klusterlet-config\") pod \"klusterlet-addon-workmgr-69f8df4d98-hwmp7\" (UID: \"0f3e1467-bcd3-44a0-b1dd-ec1b2cadf2f5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69f8df4d98-hwmp7" Sep 09 20:53:26.899246 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.899216 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f3e1467-bcd3-44a0-b1dd-ec1b2cadf2f5-tmp\") pod \"klusterlet-addon-workmgr-69f8df4d98-hwmp7\" (UID: \"0f3e1467-bcd3-44a0-b1dd-ec1b2cadf2f5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69f8df4d98-hwmp7" Sep 09 20:53:26.903284 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.903234 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0f3e1467-bcd3-44a0-b1dd-ec1b2cadf2f5-klusterlet-config\") pod \"klusterlet-addon-workmgr-69f8df4d98-hwmp7\" (UID: \"0f3e1467-bcd3-44a0-b1dd-ec1b2cadf2f5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69f8df4d98-hwmp7" Sep 09 20:53:26.911016 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.910972 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5qtl\" (UniqueName: \"kubernetes.io/projected/0f3e1467-bcd3-44a0-b1dd-ec1b2cadf2f5-kube-api-access-l5qtl\") pod \"klusterlet-addon-workmgr-69f8df4d98-hwmp7\" (UID: \"0f3e1467-bcd3-44a0-b1dd-ec1b2cadf2f5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69f8df4d98-hwmp7" Sep 09 20:53:26.946248 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.946072 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:26.995105 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.995069 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69f8df4d98-hwmp7" Sep 09 20:53:26.999665 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:26.999630 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvwrv\" (UniqueName: \"kubernetes.io/projected/1ee25625-54d3-4ac1-912c-42f78f271480-kube-api-access-pvwrv\") pod \"downloads-85d98c998f-jrfbb\" (UID: \"1ee25625-54d3-4ac1-912c-42f78f271480\") " pod="openshift-console/downloads-85d98c998f-jrfbb" Sep 09 20:53:27.009300 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:27.009272 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvwrv\" (UniqueName: \"kubernetes.io/projected/1ee25625-54d3-4ac1-912c-42f78f271480-kube-api-access-pvwrv\") pod \"downloads-85d98c998f-jrfbb\" (UID: \"1ee25625-54d3-4ac1-912c-42f78f271480\") " pod="openshift-console/downloads-85d98c998f-jrfbb" Sep 09 20:53:27.045089 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:27.045052 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:27.122509 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:27.122472 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-jrfbb" Sep 09 20:53:27.145757 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:27.145602 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:27.247730 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:27.247695 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:27.304609 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:27.304575 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69f8df4d98-hwmp7"] Sep 09 20:53:27.309315 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:53:27.309285 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3e1467_bcd3_44a0_b1dd_ec1b2cadf2f5.slice/crio-51f0376ec8303ecbdda38d8a983a4f9e098beb5eb002054b49109d239f64645b WatchSource:0}: Error finding container 51f0376ec8303ecbdda38d8a983a4f9e098beb5eb002054b49109d239f64645b: Status 404 returned error can't find the container with id 51f0376ec8303ecbdda38d8a983a4f9e098beb5eb002054b49109d239f64645b Sep 09 20:53:27.394038 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:27.393999 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:27.437781 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:27.437745 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-jrfbb"] Sep 09 20:53:27.441645 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:53:27.441619 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee25625_54d3_4ac1_912c_42f78f271480.slice/crio-6e7bea9ffd84937bd105bda9aa610569acdbf5c67a781fd70105652a6a337145 WatchSource:0}: Error finding container 6e7bea9ffd84937bd105bda9aa610569acdbf5c67a781fd70105652a6a337145: Status 404 returned error can't find the container with id 6e7bea9ffd84937bd105bda9aa610569acdbf5c67a781fd70105652a6a337145 Sep 09 20:53:27.465451 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:27.465368 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:27.821774 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:27.821663 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-jrfbb" event={"ID":"1ee25625-54d3-4ac1-912c-42f78f271480","Type":"ContainerStarted","Data":"6e7bea9ffd84937bd105bda9aa610569acdbf5c67a781fd70105652a6a337145"} Sep 09 20:53:27.823238 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:27.823205 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69f8df4d98-hwmp7" event={"ID":"0f3e1467-bcd3-44a0-b1dd-ec1b2cadf2f5","Type":"ContainerStarted","Data":"51f0376ec8303ecbdda38d8a983a4f9e098beb5eb002054b49109d239f64645b"} Sep 09 20:53:28.332965 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:28.332908 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:28.429534 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:28.429494 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:28.521612 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:28.521570 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:29.066942 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:29.066900 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:29.843721 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:29.843626 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hrmkf" event={"ID":"92eda8aa-9402-4df8-b58e-5468b16a888e","Type":"ContainerStarted","Data":"1f56a954c0df935ae6d0b779b1864e1a20ea0fbac2b50be0a3e725a2041af4ae"} Sep 09 20:53:30.028095 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:30.027720 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95fc21f9-62ed-41e4-9609-28ffab20ab40-service-ca-bundle\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:53:30.028095 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:30.027797 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8d11438c-9ef0-491b-9813-36b419613f92-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cljj5\" (UID: \"8d11438c-9ef0-491b-9813-36b419613f92\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cljj5" Sep 09 20:53:30.028648 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:30.028621 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95fc21f9-62ed-41e4-9609-28ffab20ab40-service-ca-bundle\") pod \"router-default-7f7bbb8cc6-ptg74\" (UID: \"95fc21f9-62ed-41e4-9609-28ffab20ab40\") " pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:53:30.031582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:30.031532 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8d11438c-9ef0-491b-9813-36b419613f92-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cljj5\" (UID: \"8d11438c-9ef0-491b-9813-36b419613f92\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cljj5" Sep 09 20:53:30.232556 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:30.232472 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-kwj46\"" Sep 09 20:53:30.240650 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:30.240617 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-cljj5" Sep 09 20:53:30.258509 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:30.258484 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qn5sb\"" Sep 09 20:53:30.266919 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:30.266885 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:53:30.543592 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:30.543462 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-cljj5"] Sep 09 20:53:30.548793 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:53:30.548743 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d11438c_9ef0_491b_9813_36b419613f92.slice/crio-ded9fbcd834f1cbce60ab344712253e3d91c028872f3bb2bdbbd2d7eb6e4895d WatchSource:0}: Error finding container ded9fbcd834f1cbce60ab344712253e3d91c028872f3bb2bdbbd2d7eb6e4895d: Status 404 returned error can't find the container with id ded9fbcd834f1cbce60ab344712253e3d91c028872f3bb2bdbbd2d7eb6e4895d Sep 09 20:53:30.567292 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:30.567212 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f7bbb8cc6-ptg74"] Sep 09 20:53:30.571631 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:53:30.571424 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95fc21f9_62ed_41e4_9609_28ffab20ab40.slice/crio-689a0f7972200e95162f7dca0bf75d44c330afc7d765c6f6e81766c52cbe01a1 WatchSource:0}: Error finding container 689a0f7972200e95162f7dca0bf75d44c330afc7d765c6f6e81766c52cbe01a1: Status 404 returned error can't find the container with id 689a0f7972200e95162f7dca0bf75d44c330afc7d765c6f6e81766c52cbe01a1 Sep 09 20:53:30.852347 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:30.852304 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hrmkf" event={"ID":"92eda8aa-9402-4df8-b58e-5468b16a888e","Type":"ContainerStarted","Data":"9b23260c7b38b61970128fb295c06e4a34b5718e9515152418434308828e6cd4"} Sep 09 20:53:30.853222 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:30.853193 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hrmkf" Sep 09 20:53:30.855329 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:30.855261 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-cljj5" event={"ID":"8d11438c-9ef0-491b-9813-36b419613f92","Type":"ContainerStarted","Data":"ded9fbcd834f1cbce60ab344712253e3d91c028872f3bb2bdbbd2d7eb6e4895d"} Sep 09 20:53:30.857495 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:30.857468 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" event={"ID":"95fc21f9-62ed-41e4-9609-28ffab20ab40","Type":"ContainerStarted","Data":"cecbfe907ba782ff7fb5cd2ab3ba65ad4fcff9b875a89714f90557484602d8bc"} Sep 09 20:53:30.857616 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:30.857519 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" event={"ID":"95fc21f9-62ed-41e4-9609-28ffab20ab40","Type":"ContainerStarted","Data":"689a0f7972200e95162f7dca0bf75d44c330afc7d765c6f6e81766c52cbe01a1"} Sep 09 20:53:30.859324 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:30.859297 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kf998" event={"ID":"0b2c736d-7746-4285-93bc-5ffd114396f1","Type":"ContainerStarted","Data":"94ab5a08221d20ad52b5f929005df4cb6f68e3b92f8d574b48bf5055edeca4b0"} Sep 09 20:53:30.879353 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:30.879160 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hrmkf" podStartSLOduration=34.370231566 podStartE2EDuration="37.879137462s" totalImagesPullingTime="3.508905896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:30.877612241 +0000 UTC m=+71.919612051" watchObservedRunningTime="2026-09-09 20:53:30.879137462 +0000 UTC m=+71.921137272" Sep 09 20:53:30.899805 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:30.899748 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" podStartSLOduration=32.899725158 podStartE2EDuration="32.899725158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:30.897960898 +0000 UTC m=+71.939960709" watchObservedRunningTime="2026-09-09 20:53:30.899725158 +0000 UTC m=+71.941724969" Sep 09 20:53:30.922435 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:30.922295 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kf998" podStartSLOduration=65.856002259 podStartE2EDuration="1m10.922274645s" totalImagesPullingTime="5.066272386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:30.922118724 +0000 UTC m=+71.964118533" watchObservedRunningTime="2026-09-09 20:53:30.922274645 +0000 UTC m=+71.964274455" Sep 09 20:53:31.184194 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:31.184107 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:31.268027 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:31.267789 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:53:31.271035 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:31.271003 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:53:31.300971 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:31.300887 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:31.866613 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:31.866556 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:53:31.867746 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:31.867723 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7f7bbb8cc6-ptg74" Sep 09 20:53:32.721723 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:32.721686 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:32.822473 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:32.822417 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:34.884151 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:34.884105 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-cljj5" event={"ID":"8d11438c-9ef0-491b-9813-36b419613f92","Type":"ContainerStarted","Data":"662623640fc314ca45aab207fbdae60155dded6631b84105cbae4aee43b1922e"} Sep 09 20:53:34.885766 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:34.885734 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69f8df4d98-hwmp7" event={"ID":"0f3e1467-bcd3-44a0-b1dd-ec1b2cadf2f5","Type":"ContainerStarted","Data":"259d132d039069bea6777b352829f9e50d3e2c6c3c9c65628c441da18aadc0ba"} Sep 09 20:53:34.885972 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:34.885955 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69f8df4d98-hwmp7" Sep 09 20:53:34.888025 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:34.887994 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69f8df4d98-hwmp7" Sep 09 20:53:34.901950 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:34.901877 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-cljj5" podStartSLOduration=33.657465415 podStartE2EDuration="36.901855159s" totalImagesPullingTime="3.244389744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:34.900099482 +0000 UTC m=+75.942099292" watchObservedRunningTime="2026-09-09 20:53:34.901855159 +0000 UTC m=+75.943854969" Sep 09 20:53:34.918144 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:34.918083 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69f8df4d98-hwmp7" podStartSLOduration=2.415468485 podStartE2EDuration="8.918057538s" totalImagesPullingTime="6.502589053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:34.916582435 +0000 UTC m=+75.958582247" watchObservedRunningTime="2026-09-09 20:53:34.918057538 +0000 UTC m=+75.960057343" Sep 09 20:53:35.044996 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.044960 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:35.146813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.146699 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:35.176665 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.176623 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9680a3c-cfee-45a8-8793-8a5d5888935b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:35.179408 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.179381 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9680a3c-cfee-45a8-8793-8a5d5888935b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kfg2r\" (UID: \"d9680a3c-cfee-45a8-8793-8a5d5888935b\") " pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:35.377727 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.377687 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bk2tb"] Sep 09 20:53:35.384596 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.384561 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bk2tb" Sep 09 20:53:35.387488 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.387272 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 09 20:53:35.387488 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.387325 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-4wc72\"" Sep 09 20:53:35.391294 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.391266 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bk2tb"] Sep 09 20:53:35.392286 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.392213 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:35.409699 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.409622 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s99rg\"" Sep 09 20:53:35.417661 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.417628 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kfg2r" Sep 09 20:53:35.480554 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.480440 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ec2d4891-a6a2-4729-8ff8-f733dd672361-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-bk2tb\" (UID: \"ec2d4891-a6a2-4729-8ff8-f733dd672361\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bk2tb" Sep 09 20:53:35.486196 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.486153 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:35.581644 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.581604 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ec2d4891-a6a2-4729-8ff8-f733dd672361-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-bk2tb\" (UID: \"ec2d4891-a6a2-4729-8ff8-f733dd672361\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bk2tb" Sep 09 20:53:35.581832 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:35.581718 2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 09 20:53:35.581832 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:35.581798 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec2d4891-a6a2-4729-8ff8-f733dd672361-tls-certificates podName:ec2d4891-a6a2-4729-8ff8-f733dd672361 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:36.081775902 +0000 UTC m=+77.123775690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/ec2d4891-a6a2-4729-8ff8-f733dd672361-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-bk2tb" (UID: "ec2d4891-a6a2-4729-8ff8-f733dd672361") : secret "prometheus-operator-admission-webhook-tls" not found Sep 09 20:53:35.704396 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.704319 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kfg2r"] Sep 09 20:53:35.708801 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:53:35.708261 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9680a3c_cfee_45a8_8793_8a5d5888935b.slice/crio-29dfb4cf6b638a5cf7346f4a98456419edacf224b810793b172ce963c901a860 WatchSource:0}: Error finding container 29dfb4cf6b638a5cf7346f4a98456419edacf224b810793b172ce963c901a860: Status 404 returned error can't find the container with id 29dfb4cf6b638a5cf7346f4a98456419edacf224b810793b172ce963c901a860 Sep 09 20:53:35.724079 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.724047 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85c765599c-tlgn6"] Sep 09 20:53:35.748010 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.747975 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85c765599c-tlgn6"] Sep 09 20:53:35.748176 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.748162 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c765599c-tlgn6" Sep 09 20:53:35.756257 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.756231 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jzlgf\"" Sep 09 20:53:35.756372 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.756278 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 09 20:53:35.756634 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.756547 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 09 20:53:35.756634 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.756559 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 09 20:53:35.756828 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.756787 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 09 20:53:35.756891 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.756875 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 09 20:53:35.821566 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.821530 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:35.883692 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.883660 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2da781ca-4940-4506-ab3f-20fdff5b4efc-console-oauth-config\") pod \"console-85c765599c-tlgn6\" (UID: \"2da781ca-4940-4506-ab3f-20fdff5b4efc\") " pod="openshift-console/console-85c765599c-tlgn6" Sep 09 20:53:35.883692 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.883695 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2da781ca-4940-4506-ab3f-20fdff5b4efc-console-config\") pod \"console-85c765599c-tlgn6\" (UID: \"2da781ca-4940-4506-ab3f-20fdff5b4efc\") " pod="openshift-console/console-85c765599c-tlgn6" Sep 09 20:53:35.883914 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.883786 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2da781ca-4940-4506-ab3f-20fdff5b4efc-oauth-serving-cert\") pod \"console-85c765599c-tlgn6\" (UID: \"2da781ca-4940-4506-ab3f-20fdff5b4efc\") " pod="openshift-console/console-85c765599c-tlgn6" Sep 09 20:53:35.883914 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.883866 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2da781ca-4940-4506-ab3f-20fdff5b4efc-service-ca\") pod \"console-85c765599c-tlgn6\" (UID: \"2da781ca-4940-4506-ab3f-20fdff5b4efc\") " pod="openshift-console/console-85c765599c-tlgn6" Sep 09 20:53:35.883914 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.883891 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2da781ca-4940-4506-ab3f-20fdff5b4efc-console-serving-cert\") pod \"console-85c765599c-tlgn6\" (UID: \"2da781ca-4940-4506-ab3f-20fdff5b4efc\") " pod="openshift-console/console-85c765599c-tlgn6" Sep 09 20:53:35.884095 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.883914 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-449qj\" (UniqueName: \"kubernetes.io/projected/2da781ca-4940-4506-ab3f-20fdff5b4efc-kube-api-access-449qj\") pod \"console-85c765599c-tlgn6\" (UID: \"2da781ca-4940-4506-ab3f-20fdff5b4efc\") " pod="openshift-console/console-85c765599c-tlgn6" Sep 09 20:53:35.892407 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.892370 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kfg2r" event={"ID":"d9680a3c-cfee-45a8-8793-8a5d5888935b","Type":"ContainerStarted","Data":"8fb647677009a3890055974042d65de4690e37c2898e5829aaceb06d5a644995"} Sep 09 20:53:35.892872 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.892444 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kfg2r" event={"ID":"d9680a3c-cfee-45a8-8793-8a5d5888935b","Type":"ContainerStarted","Data":"29dfb4cf6b638a5cf7346f4a98456419edacf224b810793b172ce963c901a860"} Sep 09 20:53:35.927039 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.926996 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:35.985195 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.985111 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2da781ca-4940-4506-ab3f-20fdff5b4efc-service-ca\") pod \"console-85c765599c-tlgn6\" (UID: \"2da781ca-4940-4506-ab3f-20fdff5b4efc\") " pod="openshift-console/console-85c765599c-tlgn6" Sep 09 20:53:35.985195 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.985171 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2da781ca-4940-4506-ab3f-20fdff5b4efc-console-serving-cert\") pod \"console-85c765599c-tlgn6\" (UID: \"2da781ca-4940-4506-ab3f-20fdff5b4efc\") " pod="openshift-console/console-85c765599c-tlgn6" Sep 09 20:53:35.985195 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.985197 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-449qj\" (UniqueName: \"kubernetes.io/projected/2da781ca-4940-4506-ab3f-20fdff5b4efc-kube-api-access-449qj\") pod \"console-85c765599c-tlgn6\" (UID: \"2da781ca-4940-4506-ab3f-20fdff5b4efc\") " pod="openshift-console/console-85c765599c-tlgn6" Sep 09 20:53:35.985476 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.985227 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2da781ca-4940-4506-ab3f-20fdff5b4efc-console-oauth-config\") pod \"console-85c765599c-tlgn6\" (UID: \"2da781ca-4940-4506-ab3f-20fdff5b4efc\") " pod="openshift-console/console-85c765599c-tlgn6" Sep 09 20:53:35.985476 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.985250 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2da781ca-4940-4506-ab3f-20fdff5b4efc-console-config\") pod \"console-85c765599c-tlgn6\" (UID: \"2da781ca-4940-4506-ab3f-20fdff5b4efc\") " pod="openshift-console/console-85c765599c-tlgn6" Sep 09 20:53:35.985476 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.985323 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2da781ca-4940-4506-ab3f-20fdff5b4efc-oauth-serving-cert\") pod \"console-85c765599c-tlgn6\" (UID: \"2da781ca-4940-4506-ab3f-20fdff5b4efc\") " pod="openshift-console/console-85c765599c-tlgn6" Sep 09 20:53:35.986107 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.986079 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2da781ca-4940-4506-ab3f-20fdff5b4efc-oauth-serving-cert\") pod \"console-85c765599c-tlgn6\" (UID: \"2da781ca-4940-4506-ab3f-20fdff5b4efc\") " pod="openshift-console/console-85c765599c-tlgn6" Sep 09 20:53:35.986107 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.986090 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2da781ca-4940-4506-ab3f-20fdff5b4efc-service-ca\") pod \"console-85c765599c-tlgn6\" (UID: \"2da781ca-4940-4506-ab3f-20fdff5b4efc\") " pod="openshift-console/console-85c765599c-tlgn6" Sep 09 20:53:35.986305 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.986286 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2da781ca-4940-4506-ab3f-20fdff5b4efc-console-config\") pod \"console-85c765599c-tlgn6\" (UID: \"2da781ca-4940-4506-ab3f-20fdff5b4efc\") " pod="openshift-console/console-85c765599c-tlgn6" Sep 09 20:53:35.989004 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.988979 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2da781ca-4940-4506-ab3f-20fdff5b4efc-console-serving-cert\") pod \"console-85c765599c-tlgn6\" (UID: \"2da781ca-4940-4506-ab3f-20fdff5b4efc\") " pod="openshift-console/console-85c765599c-tlgn6" Sep 09 20:53:35.989133 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.988987 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2da781ca-4940-4506-ab3f-20fdff5b4efc-console-oauth-config\") pod \"console-85c765599c-tlgn6\" (UID: \"2da781ca-4940-4506-ab3f-20fdff5b4efc\") " pod="openshift-console/console-85c765599c-tlgn6" Sep 09 20:53:35.994450 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:35.994426 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-449qj\" (UniqueName: \"kubernetes.io/projected/2da781ca-4940-4506-ab3f-20fdff5b4efc-kube-api-access-449qj\") pod \"console-85c765599c-tlgn6\" (UID: \"2da781ca-4940-4506-ab3f-20fdff5b4efc\") " pod="openshift-console/console-85c765599c-tlgn6" Sep 09 20:53:36.086173 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:36.086130 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ec2d4891-a6a2-4729-8ff8-f733dd672361-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-bk2tb\" (UID: \"ec2d4891-a6a2-4729-8ff8-f733dd672361\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bk2tb" Sep 09 20:53:36.086365 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:36.086268 2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 09 20:53:36.086365 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:36.086342 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec2d4891-a6a2-4729-8ff8-f733dd672361-tls-certificates podName:ec2d4891-a6a2-4729-8ff8-f733dd672361 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:37.086320531 +0000 UTC m=+78.128320321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/ec2d4891-a6a2-4729-8ff8-f733dd672361-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-bk2tb" (UID: "ec2d4891-a6a2-4729-8ff8-f733dd672361") : secret "prometheus-operator-admission-webhook-tls" not found Sep 09 20:53:36.104485 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:36.104254 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c765599c-tlgn6" Sep 09 20:53:36.171365 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:36.171318 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:36.415534 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:36.415332 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85c765599c-tlgn6"] Sep 09 20:53:36.418386 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:53:36.418354 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da781ca_4940_4506_ab3f_20fdff5b4efc.slice/crio-9d818d3b1849e510a74a4d0ce65d75b21aef752aebaf6422ab464e870e14003e WatchSource:0}: Error finding container 9d818d3b1849e510a74a4d0ce65d75b21aef752aebaf6422ab464e870e14003e: Status 404 returned error can't find the container with id 9d818d3b1849e510a74a4d0ce65d75b21aef752aebaf6422ab464e870e14003e Sep 09 20:53:36.824781 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:36.824699 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:36.890983 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:36.890949 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:36.898347 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:36.898299 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c765599c-tlgn6" event={"ID":"2da781ca-4940-4506-ab3f-20fdff5b4efc","Type":"ContainerStarted","Data":"9d818d3b1849e510a74a4d0ce65d75b21aef752aebaf6422ab464e870e14003e"} Sep 09 20:53:37.095147 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:37.095116 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ec2d4891-a6a2-4729-8ff8-f733dd672361-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-bk2tb\" (UID: \"ec2d4891-a6a2-4729-8ff8-f733dd672361\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bk2tb" Sep 09 20:53:37.098334 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:37.098303 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ec2d4891-a6a2-4729-8ff8-f733dd672361-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-bk2tb\" (UID: \"ec2d4891-a6a2-4729-8ff8-f733dd672361\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bk2tb" Sep 09 20:53:37.197242 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:37.197111 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bk2tb" Sep 09 20:53:37.529671 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:37.529573 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bk2tb"] Sep 09 20:53:37.533950 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:53:37.533619 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2d4891_a6a2_4729_8ff8_f733dd672361.slice/crio-eb65b179a4b643ad753ff9fcdc07b228aadd48131862584b055b4abd18f44eae WatchSource:0}: Error finding container eb65b179a4b643ad753ff9fcdc07b228aadd48131862584b055b4abd18f44eae: Status 404 returned error can't find the container with id eb65b179a4b643ad753ff9fcdc07b228aadd48131862584b055b4abd18f44eae Sep 09 20:53:37.904971 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:37.904906 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bk2tb" event={"ID":"ec2d4891-a6a2-4729-8ff8-f733dd672361","Type":"ContainerStarted","Data":"eb65b179a4b643ad753ff9fcdc07b228aadd48131862584b055b4abd18f44eae"} Sep 09 20:53:37.907543 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:37.907513 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kfg2r" event={"ID":"d9680a3c-cfee-45a8-8793-8a5d5888935b","Type":"ContainerStarted","Data":"61e60feacfb32c2f24f5b0113e81896b543e37f85ca8a0ffdd75a8b311fdbae0"} Sep 09 20:53:40.442954 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:40.442900 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:41.430951 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:41.430788 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:41.534780 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:41.534741 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:41.631649 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:41.631571 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:41.874295 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:41.874259 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hrmkf" Sep 09 20:53:42.155277 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:42.155167 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:42.244421 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:42.244385 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:42.782330 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:42.782281 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:42.867080 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:42.867046 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:43.629038 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:43.628922 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:43.698677 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:43.698636 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:46.263673 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:46.263638 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:46.354165 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:46.354099 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:47.955064 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:47.954579 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kfg2r" event={"ID":"d9680a3c-cfee-45a8-8793-8a5d5888935b","Type":"ContainerStarted","Data":"51de67e79976aa1ad5fe08ebb7abb62d303f2d6945d89e0556c57ca427f8b2e4"} Sep 09 20:53:47.957111 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:47.957074 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-jrfbb" event={"ID":"1ee25625-54d3-4ac1-912c-42f78f271480","Type":"ContainerStarted","Data":"3bef12752545f0ee080aeab6bf3282eafc137046e1f702b00f42c0e4b096519b"} Sep 09 20:53:47.957330 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:47.957303 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-jrfbb" Sep 09 20:53:47.959048 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:47.959019 2562 patch_prober.go:28] interesting pod/downloads-85d98c998f-jrfbb container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.15:8080/\": dial tcp 10.133.0.15:8080: connect: connection refused" start-of-body= Sep 09 20:53:47.959169 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:47.959072 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-85d98c998f-jrfbb" podUID="1ee25625-54d3-4ac1-912c-42f78f271480" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.15:8080/\": dial tcp 10.133.0.15:8080: connect: connection refused" Sep 09 20:53:47.959265 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:47.959021 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c765599c-tlgn6" event={"ID":"2da781ca-4940-4506-ab3f-20fdff5b4efc","Type":"ContainerStarted","Data":"315ddea3d4d1bf8b717aaab09705a0b3a42ebd883c0750eb038da23bf31b3ade"} Sep 09 20:53:47.960982 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:47.960951 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bk2tb" event={"ID":"ec2d4891-a6a2-4729-8ff8-f733dd672361","Type":"ContainerStarted","Data":"ee501c87905cae6d1cecd56d4beef2dbf75fee9a29c72db4d84b781c8c338104"} Sep 09 20:53:47.961174 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:47.961120 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bk2tb" Sep 09 20:53:47.965723 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:47.965701 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bk2tb" Sep 09 20:53:47.989765 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:47.989706 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kfg2r" podStartSLOduration=33.24582871 podStartE2EDuration="44.989688244s" totalImagesPullingTime="11.743859534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:53:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:47.989175837 +0000 UTC m=+89.031175647" watchObservedRunningTime="2026-09-09 20:53:47.989688244 +0000 UTC m=+89.031688054" Sep 09 20:53:48.022353 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.022283 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-bk2tb" podStartSLOduration=2.804281073 podStartE2EDuration="13.022261127s" totalImagesPullingTime="10.217980054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:48.021844215 +0000 UTC m=+89.063844025" watchObservedRunningTime="2026-09-09 20:53:48.022261127 +0000 UTC m=+89.064260938" Sep 09 20:53:48.047282 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.047168 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-jrfbb" podStartSLOduration=1.6897866750000001 podStartE2EDuration="22.047147506s" totalImagesPullingTime="20.357360831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:48.046383904 +0000 UTC m=+89.088383715" watchObservedRunningTime="2026-09-09 20:53:48.047147506 +0000 UTC m=+89.089147296" Sep 09 20:53:48.073142 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.073073 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85c765599c-tlgn6" podStartSLOduration=1.739116706 podStartE2EDuration="13.073048987s" totalImagesPullingTime="11.333932281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:48.072117104 +0000 UTC m=+89.114116939" watchObservedRunningTime="2026-09-09 20:53:48.073048987 +0000 UTC m=+89.115048797" Sep 09 20:53:48.440268 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.440211 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:48.471939 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.471875 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-p942p"] Sep 09 20:53:48.475359 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.475328 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-p942p" Sep 09 20:53:48.478035 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.478002 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 09 20:53:48.478589 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.478550 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 09 20:53:48.478864 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.478816 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 09 20:53:48.478864 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.478849 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 09 20:53:48.479194 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.479176 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qxl4m\"" Sep 09 20:53:48.479290 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.479223 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 09 20:53:48.489911 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.489881 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-p942p"] Sep 09 20:53:48.521511 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.521474 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:48.592383 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.592334 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20d7ee39-f1be-4d32-9f91-9015bc14779f-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-p942p\" (UID: \"20d7ee39-f1be-4d32-9f91-9015bc14779f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-p942p" Sep 09 20:53:48.592596 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.592480 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/20d7ee39-f1be-4d32-9f91-9015bc14779f-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-p942p\" (UID: \"20d7ee39-f1be-4d32-9f91-9015bc14779f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-p942p" Sep 09 20:53:48.592596 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.592513 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20d7ee39-f1be-4d32-9f91-9015bc14779f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-p942p\" (UID: \"20d7ee39-f1be-4d32-9f91-9015bc14779f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-p942p" Sep 09 20:53:48.592596 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.592551 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7b7p\" (UniqueName: \"kubernetes.io/projected/20d7ee39-f1be-4d32-9f91-9015bc14779f-kube-api-access-n7b7p\") pod \"prometheus-operator-6f4946db56-p942p\" (UID: \"20d7ee39-f1be-4d32-9f91-9015bc14779f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-p942p" Sep 09 20:53:48.693582 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.693492 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20d7ee39-f1be-4d32-9f91-9015bc14779f-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-p942p\" (UID: \"20d7ee39-f1be-4d32-9f91-9015bc14779f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-p942p" Sep 09 20:53:48.693754 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.693594 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/20d7ee39-f1be-4d32-9f91-9015bc14779f-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-p942p\" (UID: \"20d7ee39-f1be-4d32-9f91-9015bc14779f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-p942p" Sep 09 20:53:48.693754 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.693626 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20d7ee39-f1be-4d32-9f91-9015bc14779f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-p942p\" (UID: \"20d7ee39-f1be-4d32-9f91-9015bc14779f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-p942p" Sep 09 20:53:48.693754 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.693660 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7b7p\" (UniqueName: \"kubernetes.io/projected/20d7ee39-f1be-4d32-9f91-9015bc14779f-kube-api-access-n7b7p\") pod \"prometheus-operator-6f4946db56-p942p\" (UID: \"20d7ee39-f1be-4d32-9f91-9015bc14779f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-p942p" Sep 09 20:53:48.694112 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:48.694088 2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 09 20:53:48.694343 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:48.694325 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20d7ee39-f1be-4d32-9f91-9015bc14779f-prometheus-operator-tls podName:20d7ee39-f1be-4d32-9f91-9015bc14779f nodeName:}" failed. No retries permitted until 2026-09-09 20:53:49.194303558 +0000 UTC m=+90.236303351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/20d7ee39-f1be-4d32-9f91-9015bc14779f-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-p942p" (UID: "20d7ee39-f1be-4d32-9f91-9015bc14779f") : secret "prometheus-operator-tls" not found Sep 09 20:53:48.694462 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.694358 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20d7ee39-f1be-4d32-9f91-9015bc14779f-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-p942p\" (UID: \"20d7ee39-f1be-4d32-9f91-9015bc14779f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-p942p" Sep 09 20:53:48.696965 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.696918 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20d7ee39-f1be-4d32-9f91-9015bc14779f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-p942p\" (UID: \"20d7ee39-f1be-4d32-9f91-9015bc14779f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-p942p" Sep 09 20:53:48.707009 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.706982 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7b7p\" (UniqueName: \"kubernetes.io/projected/20d7ee39-f1be-4d32-9f91-9015bc14779f-kube-api-access-n7b7p\") pod \"prometheus-operator-6f4946db56-p942p\" (UID: \"20d7ee39-f1be-4d32-9f91-9015bc14779f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-p942p" Sep 09 20:53:48.801750 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.801711 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:48.918787 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.918746 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:48.978345 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:48.978270 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-jrfbb" Sep 09 20:53:49.036353 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:49.036322 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:49.198876 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:49.198820 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/20d7ee39-f1be-4d32-9f91-9015bc14779f-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-p942p\" (UID: \"20d7ee39-f1be-4d32-9f91-9015bc14779f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-p942p" Sep 09 20:53:49.201811 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:49.201781 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/20d7ee39-f1be-4d32-9f91-9015bc14779f-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-p942p\" (UID: \"20d7ee39-f1be-4d32-9f91-9015bc14779f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-p942p" Sep 09 20:53:49.388357 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:49.388316 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-p942p" Sep 09 20:53:49.707149 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:49.707111 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-p942p"] Sep 09 20:53:49.710176 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:53:49.710148 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d7ee39_f1be_4d32_9f91_9015bc14779f.slice/crio-bd1b0221c7f121b7948e4d75e4124f7366e088b65803d70d296b7027496472e6 WatchSource:0}: Error finding container bd1b0221c7f121b7948e4d75e4124f7366e088b65803d70d296b7027496472e6: Status 404 returned error can't find the container with id bd1b0221c7f121b7948e4d75e4124f7366e088b65803d70d296b7027496472e6 Sep 09 20:53:49.971388 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:49.971299 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-p942p" event={"ID":"20d7ee39-f1be-4d32-9f91-9015bc14779f","Type":"ContainerStarted","Data":"bd1b0221c7f121b7948e4d75e4124f7366e088b65803d70d296b7027496472e6"} Sep 09 20:53:51.864543 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:51.864499 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:51.972810 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:51.972676 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:51.982837 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:51.982792 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-p942p" event={"ID":"20d7ee39-f1be-4d32-9f91-9015bc14779f","Type":"ContainerStarted","Data":"70796e5980c055c0cc499e94951ef8ef46fd9220fedd23cb8bfd6a42a7e00f1c"} Sep 09 20:53:51.983041 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:51.982848 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-p942p" event={"ID":"20d7ee39-f1be-4d32-9f91-9015bc14779f","Type":"ContainerStarted","Data":"65a839012c094f4da52ca5f2f4070a4f28ec36c75ffcfbfebbfe687e747eab04"} Sep 09 20:53:52.002558 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:52.002491 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-p942p" podStartSLOduration=2.06811542 podStartE2EDuration="4.002469692s" totalImagesPullingTime="1.934354272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:52.001572321 +0000 UTC m=+93.043572155" watchObservedRunningTime="2026-09-09 20:53:52.002469692 +0000 UTC m=+93.044469502" Sep 09 20:53:52.064905 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:52.064867 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:52.345258 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:52.345222 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:52.446571 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:52.446531 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:53.592978 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:53.592924 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:53.775504 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:53.775464 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85c765599c-tlgn6"] Sep 09 20:53:53.832742 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:53.832698 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:53.937680 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:53.937599 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-b5v84"] Sep 09 20:53:53.950888 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:53.950853 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:53.980028 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:53.979988 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-b5v84"] Sep 09 20:53:53.980210 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:53.980123 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" Sep 09 20:53:53.984214 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:53.984186 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-fzvfp\"" Sep 09 20:53:53.984375 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:53.984186 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 09 20:53:53.984375 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:53.984293 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 09 20:53:53.984559 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:53.984535 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 09 20:53:53.985883 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:53.985803 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lptbr"] Sep 09 20:53:54.012349 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.012316 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.017859 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.017358 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v2wxl\"" Sep 09 20:53:54.017859 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.017407 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 09 20:53:54.017859 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.017362 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 09 20:53:54.017859 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.017695 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 09 20:53:54.052624 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.051689 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:54.143311 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.143252 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8b7eabc-e75b-436f-be68-413fe8ba86d0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.143311 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.143300 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ca1e078-17b2-4050-ba7b-202df948a805-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-b5v84\" (UID: \"0ca1e078-17b2-4050-ba7b-202df948a805\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" Sep 09 20:53:54.143600 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.143333 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ca1e078-17b2-4050-ba7b-202df948a805-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-b5v84\" (UID: \"0ca1e078-17b2-4050-ba7b-202df948a805\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" Sep 09 20:53:54.143600 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.143361 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8b7eabc-e75b-436f-be68-413fe8ba86d0-sys\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.143600 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.143404 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8b7eabc-e75b-436f-be68-413fe8ba86d0-root\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.143600 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.143429 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0ca1e078-17b2-4050-ba7b-202df948a805-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-b5v84\" (UID: \"0ca1e078-17b2-4050-ba7b-202df948a805\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" Sep 09 20:53:54.143600 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.143454 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8b7eabc-e75b-436f-be68-413fe8ba86d0-node-exporter-textfile\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.143600 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.143485 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8b7eabc-e75b-436f-be68-413fe8ba86d0-node-exporter-wtmp\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.143600 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.143508 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ca1e078-17b2-4050-ba7b-202df948a805-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-b5v84\" (UID: \"0ca1e078-17b2-4050-ba7b-202df948a805\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" Sep 09 20:53:54.143600 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.143546 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0ca1e078-17b2-4050-ba7b-202df948a805-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-b5v84\" (UID: \"0ca1e078-17b2-4050-ba7b-202df948a805\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" Sep 09 20:53:54.143600 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.143574 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmpxl\" (UniqueName: \"kubernetes.io/projected/a8b7eabc-e75b-436f-be68-413fe8ba86d0-kube-api-access-jmpxl\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.144044 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.143606 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8b7eabc-e75b-436f-be68-413fe8ba86d0-node-exporter-tls\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.144044 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.143640 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8b7eabc-e75b-436f-be68-413fe8ba86d0-node-exporter-accelerators-collector-config\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.144044 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.143664 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxtsm\" (UniqueName: \"kubernetes.io/projected/0ca1e078-17b2-4050-ba7b-202df948a805-kube-api-access-xxtsm\") pod \"kube-state-metrics-74cb57df7-b5v84\" (UID: \"0ca1e078-17b2-4050-ba7b-202df948a805\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" Sep 09 20:53:54.144044 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.143690 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8b7eabc-e75b-436f-be68-413fe8ba86d0-metrics-client-ca\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.157106 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.157065 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:54.245038 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.244925 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8b7eabc-e75b-436f-be68-413fe8ba86d0-root\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.245038 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.244992 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0ca1e078-17b2-4050-ba7b-202df948a805-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-b5v84\" (UID: \"0ca1e078-17b2-4050-ba7b-202df948a805\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" Sep 09 20:53:54.245038 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.245016 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8b7eabc-e75b-436f-be68-413fe8ba86d0-node-exporter-textfile\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.245320 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.245044 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8b7eabc-e75b-436f-be68-413fe8ba86d0-node-exporter-wtmp\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.245320 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.245056 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8b7eabc-e75b-436f-be68-413fe8ba86d0-root\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.245320 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.245073 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ca1e078-17b2-4050-ba7b-202df948a805-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-b5v84\" (UID: \"0ca1e078-17b2-4050-ba7b-202df948a805\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" Sep 09 20:53:54.245320 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.245125 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0ca1e078-17b2-4050-ba7b-202df948a805-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-b5v84\" (UID: \"0ca1e078-17b2-4050-ba7b-202df948a805\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" Sep 09 20:53:54.245320 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.245159 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmpxl\" (UniqueName: \"kubernetes.io/projected/a8b7eabc-e75b-436f-be68-413fe8ba86d0-kube-api-access-jmpxl\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.245320 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.245192 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8b7eabc-e75b-436f-be68-413fe8ba86d0-node-exporter-tls\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.245320 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.245225 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8b7eabc-e75b-436f-be68-413fe8ba86d0-node-exporter-accelerators-collector-config\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.245320 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.245250 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxtsm\" (UniqueName: \"kubernetes.io/projected/0ca1e078-17b2-4050-ba7b-202df948a805-kube-api-access-xxtsm\") pod \"kube-state-metrics-74cb57df7-b5v84\" (UID: \"0ca1e078-17b2-4050-ba7b-202df948a805\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" Sep 09 20:53:54.245320 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.245277 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8b7eabc-e75b-436f-be68-413fe8ba86d0-metrics-client-ca\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.245320 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.245311 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8b7eabc-e75b-436f-be68-413fe8ba86d0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.245797 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.245337 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ca1e078-17b2-4050-ba7b-202df948a805-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-b5v84\" (UID: \"0ca1e078-17b2-4050-ba7b-202df948a805\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" Sep 09 20:53:54.245797 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.245369 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ca1e078-17b2-4050-ba7b-202df948a805-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-b5v84\" (UID: \"0ca1e078-17b2-4050-ba7b-202df948a805\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" Sep 09 20:53:54.245797 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.245399 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8b7eabc-e75b-436f-be68-413fe8ba86d0-sys\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.245797 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.245398 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8b7eabc-e75b-436f-be68-413fe8ba86d0-node-exporter-textfile\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.245797 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.245458 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8b7eabc-e75b-436f-be68-413fe8ba86d0-sys\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.245797 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.245598 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8b7eabc-e75b-436f-be68-413fe8ba86d0-node-exporter-wtmp\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.246105 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.245796 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0ca1e078-17b2-4050-ba7b-202df948a805-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-b5v84\" (UID: \"0ca1e078-17b2-4050-ba7b-202df948a805\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" Sep 09 20:53:54.246782 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.246751 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8b7eabc-e75b-436f-be68-413fe8ba86d0-node-exporter-accelerators-collector-config\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.247027 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.247002 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0ca1e078-17b2-4050-ba7b-202df948a805-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-b5v84\" (UID: \"0ca1e078-17b2-4050-ba7b-202df948a805\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" Sep 09 20:53:54.247643 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.247615 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8b7eabc-e75b-436f-be68-413fe8ba86d0-metrics-client-ca\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.247739 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:54.247716 2562 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 09 20:53:54.247795 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:53:54.247783 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ca1e078-17b2-4050-ba7b-202df948a805-kube-state-metrics-tls podName:0ca1e078-17b2-4050-ba7b-202df948a805 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:54.747759591 +0000 UTC m=+95.789759378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/0ca1e078-17b2-4050-ba7b-202df948a805-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-b5v84" (UID: "0ca1e078-17b2-4050-ba7b-202df948a805") : secret "kube-state-metrics-tls" not found Sep 09 20:53:54.248165 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.248144 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ca1e078-17b2-4050-ba7b-202df948a805-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-b5v84\" (UID: \"0ca1e078-17b2-4050-ba7b-202df948a805\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" Sep 09 20:53:54.249524 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.249494 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8b7eabc-e75b-436f-be68-413fe8ba86d0-node-exporter-tls\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.250896 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.250853 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ca1e078-17b2-4050-ba7b-202df948a805-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-b5v84\" (UID: \"0ca1e078-17b2-4050-ba7b-202df948a805\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" Sep 09 20:53:54.251395 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.251351 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8b7eabc-e75b-436f-be68-413fe8ba86d0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.256814 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.256791 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxtsm\" (UniqueName: \"kubernetes.io/projected/0ca1e078-17b2-4050-ba7b-202df948a805-kube-api-access-xxtsm\") pod \"kube-state-metrics-74cb57df7-b5v84\" (UID: \"0ca1e078-17b2-4050-ba7b-202df948a805\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" Sep 09 20:53:54.259253 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.259230 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmpxl\" (UniqueName: \"kubernetes.io/projected/a8b7eabc-e75b-436f-be68-413fe8ba86d0-kube-api-access-jmpxl\") pod \"node-exporter-lptbr\" (UID: \"a8b7eabc-e75b-436f-be68-413fe8ba86d0\") " pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.328422 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.328381 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lptbr" Sep 09 20:53:54.369161 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:53:54.369115 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b7eabc_e75b_436f_be68_413fe8ba86d0.slice/crio-045614a10723f218d070ee1bff1db2222b23ecdc966cc92cdf36031a1c05577c WatchSource:0}: Error finding container 045614a10723f218d070ee1bff1db2222b23ecdc966cc92cdf36031a1c05577c: Status 404 returned error can't find the container with id 045614a10723f218d070ee1bff1db2222b23ecdc966cc92cdf36031a1c05577c Sep 09 20:53:54.387733 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.387696 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:54.483749 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.483712 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:54.587360 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.587325 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:54.750479 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.750400 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ca1e078-17b2-4050-ba7b-202df948a805-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-b5v84\" (UID: \"0ca1e078-17b2-4050-ba7b-202df948a805\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" Sep 09 20:53:54.755272 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.753569 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ca1e078-17b2-4050-ba7b-202df948a805-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-b5v84\" (UID: \"0ca1e078-17b2-4050-ba7b-202df948a805\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" Sep 09 20:53:54.788163 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.788122 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:54.876209 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.876108 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:54.896167 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:54.895089 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" Sep 09 20:53:55.019267 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:55.018323 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lptbr" event={"ID":"a8b7eabc-e75b-436f-be68-413fe8ba86d0","Type":"ContainerStarted","Data":"045614a10723f218d070ee1bff1db2222b23ecdc966cc92cdf36031a1c05577c"} Sep 09 20:53:55.119148 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:55.119107 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:55.200015 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:55.198347 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:55.208869 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:55.208823 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-b5v84"] Sep 09 20:53:55.214628 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:53:55.214592 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca1e078_17b2_4050_ba7b_202df948a805.slice/crio-055e0fd7ccaba65dd9dc34942e433727408d73053605eb823746a43368850d44 WatchSource:0}: Error finding container 055e0fd7ccaba65dd9dc34942e433727408d73053605eb823746a43368850d44: Status 404 returned error can't find the container with id 055e0fd7ccaba65dd9dc34942e433727408d73053605eb823746a43368850d44 Sep 09 20:53:55.482496 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:55.482400 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:55.808260 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:55.807343 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bbvk6" Sep 09 20:53:56.027996 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:56.027961 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" event={"ID":"0ca1e078-17b2-4050-ba7b-202df948a805","Type":"ContainerStarted","Data":"055e0fd7ccaba65dd9dc34942e433727408d73053605eb823746a43368850d44"} Sep 09 20:53:56.105142 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:56.105098 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85c765599c-tlgn6" Sep 09 20:53:56.810860 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:56.810748 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:56.912197 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:56.912131 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:57.035104 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:57.035065 2562 generic.go:356] "Generic (PLEG): container finished" podID="a8b7eabc-e75b-436f-be68-413fe8ba86d0" containerID="318959b6cfec733cc15b2d2a41ee28e71db731fd8f8fdcea3d3841da49a47c24" exitCode=0 Sep 09 20:53:57.035308 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:57.035156 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lptbr" event={"ID":"a8b7eabc-e75b-436f-be68-413fe8ba86d0","Type":"ContainerDied","Data":"318959b6cfec733cc15b2d2a41ee28e71db731fd8f8fdcea3d3841da49a47c24"} Sep 09 20:53:57.386206 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:57.386115 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:57.487039 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:57.487002 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:58.041047 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.040912 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" event={"ID":"0ca1e078-17b2-4050-ba7b-202df948a805","Type":"ContainerStarted","Data":"1d6d52458707884cffc920b4bdc7ae5bf7afb5cfd469c3e59fe2dfaaa3d51509"} Sep 09 20:53:58.041477 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.041049 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" event={"ID":"0ca1e078-17b2-4050-ba7b-202df948a805","Type":"ContainerStarted","Data":"df7b4f41b8087e59554350581c06ab2ff8f1ce62e0d7cb0613c761218a8fbc87"} Sep 09 20:53:58.041477 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.041079 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" event={"ID":"0ca1e078-17b2-4050-ba7b-202df948a805","Type":"ContainerStarted","Data":"edf908167957d76a22ef789260229fefcef8521f547741dcb0422c1d83408dd5"} Sep 09 20:53:58.043195 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.043164 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lptbr" event={"ID":"a8b7eabc-e75b-436f-be68-413fe8ba86d0","Type":"ContainerStarted","Data":"b9d8d067535fa6e946537bd79aa435a74e5a52d245719ba1530639ed0b7e06fe"} Sep 09 20:53:58.043324 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.043214 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lptbr" event={"ID":"a8b7eabc-e75b-436f-be68-413fe8ba86d0","Type":"ContainerStarted","Data":"b7b4ad3f33728cdd9d2d637051855263e77d525cb9aedd7d5aac2d57c973dc99"} Sep 09 20:53:58.065282 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.065227 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-b5v84" podStartSLOduration=2.778543334 podStartE2EDuration="5.065209826s" totalImagesPullingTime="2.286666492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:58.064530609 +0000 UTC m=+99.106530418" watchObservedRunningTime="2026-09-09 20:53:58.065209826 +0000 UTC m=+99.107209634" Sep 09 20:53:58.088659 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.088602 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lptbr" podStartSLOduration=3.250929349 podStartE2EDuration="5.088584275s" totalImagesPullingTime="1.73803954s" totalInitContainerRuntime="99.615386ms" isStatefulPod=true podCreationTimestamp="2026-09-09 20:53:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:58.087898616 +0000 UTC m=+99.129898451" watchObservedRunningTime="2026-09-09 20:53:58.088584275 +0000 UTC m=+99.130584111" Sep 09 20:53:58.196760 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.196727 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-646bcb899d-882pt"] Sep 09 20:53:58.212607 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.212573 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-646bcb899d-882pt"] Sep 09 20:53:58.212799 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.212701 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:53:58.216087 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.216061 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 09 20:53:58.216233 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.216191 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7688m92vj007i\"" Sep 09 20:53:58.216307 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.216230 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ctlz4\"" Sep 09 20:53:58.216307 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.216215 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 09 20:53:58.216609 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.216586 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 09 20:53:58.216609 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.216600 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 09 20:53:58.387096 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.387061 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708-client-ca-bundle\") pod \"metrics-server-646bcb899d-882pt\" (UID: \"d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708\") " pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:53:58.387297 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.387108 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708-audit-log\") pod \"metrics-server-646bcb899d-882pt\" (UID: \"d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708\") " pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:53:58.387297 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.387170 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6psf\" (UniqueName: \"kubernetes.io/projected/d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708-kube-api-access-n6psf\") pod \"metrics-server-646bcb899d-882pt\" (UID: \"d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708\") " pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:53:58.387297 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.387209 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708-metrics-server-audit-profiles\") pod \"metrics-server-646bcb899d-882pt\" (UID: \"d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708\") " pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:53:58.387297 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.387248 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708-secret-metrics-server-tls\") pod \"metrics-server-646bcb899d-882pt\" (UID: \"d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708\") " pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:53:58.387297 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.387273 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708-secret-metrics-server-client-certs\") pod \"metrics-server-646bcb899d-882pt\" (UID: \"d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708\") " pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:53:58.387488 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.387305 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-646bcb899d-882pt\" (UID: \"d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708\") " pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:53:58.488395 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.488356 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6psf\" (UniqueName: \"kubernetes.io/projected/d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708-kube-api-access-n6psf\") pod \"metrics-server-646bcb899d-882pt\" (UID: \"d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708\") " pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:53:58.488591 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.488411 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708-metrics-server-audit-profiles\") pod \"metrics-server-646bcb899d-882pt\" (UID: \"d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708\") " pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:53:58.488591 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.488448 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708-secret-metrics-server-tls\") pod \"metrics-server-646bcb899d-882pt\" (UID: \"d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708\") " pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:53:58.488591 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.488478 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708-secret-metrics-server-client-certs\") pod \"metrics-server-646bcb899d-882pt\" (UID: \"d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708\") " pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:53:58.488591 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.488521 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-646bcb899d-882pt\" (UID: \"d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708\") " pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:53:58.488591 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.488557 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708-client-ca-bundle\") pod \"metrics-server-646bcb899d-882pt\" (UID: \"d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708\") " pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:53:58.488591 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.488590 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708-audit-log\") pod \"metrics-server-646bcb899d-882pt\" (UID: \"d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708\") " pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:53:58.489077 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.489044 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708-audit-log\") pod \"metrics-server-646bcb899d-882pt\" (UID: \"d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708\") " pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:53:58.489401 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.489380 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-646bcb899d-882pt\" (UID: \"d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708\") " pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:53:58.489673 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.489648 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708-metrics-server-audit-profiles\") pod \"metrics-server-646bcb899d-882pt\" (UID: \"d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708\") " pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:53:58.491037 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.491017 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708-secret-metrics-server-tls\") pod \"metrics-server-646bcb899d-882pt\" (UID: \"d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708\") " pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:53:58.491145 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.491119 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708-client-ca-bundle\") pod \"metrics-server-646bcb899d-882pt\" (UID: \"d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708\") " pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:53:58.491310 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.491289 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708-secret-metrics-server-client-certs\") pod \"metrics-server-646bcb899d-882pt\" (UID: \"d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708\") " pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:53:58.496589 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.496563 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6psf\" (UniqueName: \"kubernetes.io/projected/d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708-kube-api-access-n6psf\") pod \"metrics-server-646bcb899d-882pt\" (UID: \"d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708\") " pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:53:58.522127 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.522097 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:53:58.663346 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.663218 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:58.684652 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.684618 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-fns2c"] Sep 09 20:53:58.710502 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.710461 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-fns2c"] Sep 09 20:53:58.710670 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.710590 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fns2c" Sep 09 20:53:58.714607 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.714582 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 09 20:53:58.714800 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.714687 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-s8xj5\"" Sep 09 20:53:58.750833 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.750798 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:58.833065 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.833032 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-646bcb899d-882pt"] Sep 09 20:53:58.836421 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:53:58.836390 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ef4dc8_92d8_4e62_8b3e_69b3c3e2f708.slice/crio-76e3aba33fbe9781821573fc8b4f3b2b8d93d116fa5e82e5d58c3cd3fc0f3b44 WatchSource:0}: Error finding container 76e3aba33fbe9781821573fc8b4f3b2b8d93d116fa5e82e5d58c3cd3fc0f3b44: Status 404 returned error can't find the container with id 76e3aba33fbe9781821573fc8b4f3b2b8d93d116fa5e82e5d58c3cd3fc0f3b44 Sep 09 20:53:58.842261 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.842234 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:58.893379 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.893339 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8m76\" (UniqueName: \"kubernetes.io/projected/1b087158-63bd-4adc-a13f-0fee6d559f51-kube-api-access-n8m76\") pod \"monitoring-plugin-85bbbbbf4-fns2c\" (UID: \"1b087158-63bd-4adc-a13f-0fee6d559f51\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fns2c" Sep 09 20:53:58.893571 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.893408 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b087158-63bd-4adc-a13f-0fee6d559f51-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-fns2c\" (UID: \"1b087158-63bd-4adc-a13f-0fee6d559f51\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fns2c" Sep 09 20:53:58.917302 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.917210 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:58.994178 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.994137 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8m76\" (UniqueName: \"kubernetes.io/projected/1b087158-63bd-4adc-a13f-0fee6d559f51-kube-api-access-n8m76\") pod \"monitoring-plugin-85bbbbbf4-fns2c\" (UID: \"1b087158-63bd-4adc-a13f-0fee6d559f51\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fns2c" Sep 09 20:53:58.994381 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.994289 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b087158-63bd-4adc-a13f-0fee6d559f51-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-fns2c\" (UID: \"1b087158-63bd-4adc-a13f-0fee6d559f51\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fns2c" Sep 09 20:53:58.996782 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:58.996754 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1b087158-63bd-4adc-a13f-0fee6d559f51-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-fns2c\" (UID: \"1b087158-63bd-4adc-a13f-0fee6d559f51\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fns2c" Sep 09 20:53:59.003197 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:59.003175 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8m76\" (UniqueName: \"kubernetes.io/projected/1b087158-63bd-4adc-a13f-0fee6d559f51-kube-api-access-n8m76\") pod \"monitoring-plugin-85bbbbbf4-fns2c\" (UID: \"1b087158-63bd-4adc-a13f-0fee6d559f51\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fns2c" Sep 09 20:53:59.018191 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:59.018163 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:53:59.022010 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:59.021985 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fns2c" Sep 09 20:53:59.048858 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:59.048823 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-646bcb899d-882pt" event={"ID":"d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708","Type":"ContainerStarted","Data":"76e3aba33fbe9781821573fc8b4f3b2b8d93d116fa5e82e5d58c3cd3fc0f3b44"} Sep 09 20:53:59.315578 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:53:59.315490 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-fns2c"] Sep 09 20:53:59.318533 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:53:59.318508 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b087158_63bd_4adc_a13f_0fee6d559f51.slice/crio-18a0e6c2e5c3d385ecbcd0d11167cfd4b3d0d01e33c1008c2400db40874bd256 WatchSource:0}: Error finding container 18a0e6c2e5c3d385ecbcd0d11167cfd4b3d0d01e33c1008c2400db40874bd256: Status 404 returned error can't find the container with id 18a0e6c2e5c3d385ecbcd0d11167cfd4b3d0d01e33c1008c2400db40874bd256 Sep 09 20:54:00.055565 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.055518 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fns2c" event={"ID":"1b087158-63bd-4adc-a13f-0fee6d559f51","Type":"ContainerStarted","Data":"18a0e6c2e5c3d385ecbcd0d11167cfd4b3d0d01e33c1008c2400db40874bd256"} Sep 09 20:54:00.089169 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.089132 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:00.096553 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.096507 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 20:54:00.120224 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.120188 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.120401 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.120262 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 20:54:00.123413 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.123203 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 09 20:54:00.123886 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.123860 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 09 20:54:00.124096 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.123952 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qjsfk\"" Sep 09 20:54:00.124189 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.123867 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 09 20:54:00.124189 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.124105 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 09 20:54:00.124326 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.124304 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 09 20:54:00.124420 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.124350 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 09 20:54:00.124605 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.124586 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 09 20:54:00.124767 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.124599 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 09 20:54:00.125156 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.125136 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6c83evuihm204\"" Sep 09 20:54:00.125255 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.125178 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 09 20:54:00.125436 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.125412 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 09 20:54:00.125648 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.125630 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 09 20:54:00.125841 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.125824 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 09 20:54:00.129189 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.129163 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 09 20:54:00.137879 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.137851 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 09 20:54:00.204617 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.204578 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v4zt\" (UniqueName: \"kubernetes.io/projected/dafe8c25-2709-4617-9725-695d19a09b1e-kube-api-access-8v4zt\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.204825 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.204635 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.204825 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.204662 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.204825 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.204747 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.205025 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.204855 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.205025 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.204894 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dafe8c25-2709-4617-9725-695d19a09b1e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.205025 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.204912 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.205025 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.204950 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dafe8c25-2709-4617-9725-695d19a09b1e-config-out\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.205025 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.204982 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.205263 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.205087 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.205263 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.205118 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.205263 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.205140 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.205263 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.205163 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.205263 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.205187 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-web-config\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.205263 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.205227 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.205518 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.205271 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-config\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.205518 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.205291 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.205518 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.205315 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.205518 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.205360 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.205518 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.205384 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.307698 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.306542 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.307698 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.306595 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.307698 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.306628 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.307698 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.306660 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.307698 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.306688 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v4zt\" (UniqueName: \"kubernetes.io/projected/dafe8c25-2709-4617-9725-695d19a09b1e-kube-api-access-8v4zt\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.307698 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.306716 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.307698 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.306742 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.307698 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.306766 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.307698 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.306803 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.307698 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.306831 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dafe8c25-2709-4617-9725-695d19a09b1e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.307698 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.306852 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.307698 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.306879 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dafe8c25-2709-4617-9725-695d19a09b1e-config-out\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.307698 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.306908 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.307698 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.306989 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.307698 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.307018 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.307698 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.307040 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.307698 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.307060 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.309129 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.307080 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-web-config\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.309129 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.307105 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.309129 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.307150 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-config\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.309129 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.308790 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.309943 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.309399 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.310354 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.310072 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.311616 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.311040 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.311616 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.311091 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.312264 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.312237 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.312840 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.312754 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.315504 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.313370 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-config\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.315504 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.315231 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dafe8c25-2709-4617-9725-695d19a09b1e-config-out\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.315504 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.315452 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.316103 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.316078 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.317818 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.317485 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.317818 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.317763 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v4zt\" (UniqueName: \"kubernetes.io/projected/dafe8c25-2709-4617-9725-695d19a09b1e-kube-api-access-8v4zt\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.318070 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.317905 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-web-config\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.318864 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.318846 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.320531 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.320490 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.320624 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.320552 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.321499 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.321481 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dafe8c25-2709-4617-9725-695d19a09b1e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.321739 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.321682 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.322612 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.322589 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.436004 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.435964 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:00.758305 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:00.758279 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 20:54:00.770965 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:54:00.770920 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddafe8c25_2709_4617_9725_695d19a09b1e.slice/crio-0f80c9e2de8c1df0e3986ab87650db6096df021fa2fa17cffc0820b0a3659b9a WatchSource:0}: Error finding container 0f80c9e2de8c1df0e3986ab87650db6096df021fa2fa17cffc0820b0a3659b9a: Status 404 returned error can't find the container with id 0f80c9e2de8c1df0e3986ab87650db6096df021fa2fa17cffc0820b0a3659b9a Sep 09 20:54:01.064346 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:01.064244 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-646bcb899d-882pt" event={"ID":"d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708","Type":"ContainerStarted","Data":"944a6127fef4249f7f3e1246a6d91df0f0757c67583f7dc8c0c9f873e2e2e3fa"} Sep 09 20:54:01.065598 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:01.065574 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dafe8c25-2709-4617-9725-695d19a09b1e","Type":"ContainerStarted","Data":"0f80c9e2de8c1df0e3986ab87650db6096df021fa2fa17cffc0820b0a3659b9a"} Sep 09 20:54:01.085421 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:01.085346 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-646bcb899d-882pt" podStartSLOduration=1.132501201 podStartE2EDuration="3.085329084s" totalImagesPullingTime="1.952827883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:01.084231978 +0000 UTC m=+102.126231787" watchObservedRunningTime="2026-09-09 20:54:01.085329084 +0000 UTC m=+102.127328975" Sep 09 20:54:01.174107 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:01.174064 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:01.279115 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:01.279078 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:03.075549 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:03.075505 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fns2c" event={"ID":"1b087158-63bd-4adc-a13f-0fee6d559f51","Type":"ContainerStarted","Data":"57a10eb3049494680f627f2a8d8f0d833112aaa7853403a217b2bcd123c65c6c"} Sep 09 20:54:03.076065 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:03.075662 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fns2c" Sep 09 20:54:03.077077 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:03.077053 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dafe8c25-2709-4617-9725-695d19a09b1e","Type":"ContainerStarted","Data":"779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed"} Sep 09 20:54:03.080744 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:03.080720 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fns2c" Sep 09 20:54:03.092999 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:03.092914 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fns2c" podStartSLOduration=2.405168796 podStartE2EDuration="5.092891354s" totalImagesPullingTime="2.687722558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:03.090865168 +0000 UTC m=+104.132864979" watchObservedRunningTime="2026-09-09 20:54:03.092891354 +0000 UTC m=+104.134891164" Sep 09 20:54:04.082842 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:04.082809 2562 generic.go:356] "Generic (PLEG): container finished" podID="dafe8c25-2709-4617-9725-695d19a09b1e" containerID="779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed" exitCode=0 Sep 09 20:54:04.083255 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:04.082890 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dafe8c25-2709-4617-9725-695d19a09b1e","Type":"ContainerDied","Data":"779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed"} Sep 09 20:54:08.109318 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:08.109150 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dafe8c25-2709-4617-9725-695d19a09b1e","Type":"ContainerStarted","Data":"0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b"} Sep 09 20:54:08.109318 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:08.109245 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dafe8c25-2709-4617-9725-695d19a09b1e","Type":"ContainerStarted","Data":"1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b"} Sep 09 20:54:09.950070 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:09.950023 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:10.050975 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:10.050916 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:10.152172 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:10.152143 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:11.124337 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:11.124301 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dafe8c25-2709-4617-9725-695d19a09b1e","Type":"ContainerStarted","Data":"9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880"} Sep 09 20:54:11.124924 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:11.124375 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dafe8c25-2709-4617-9725-695d19a09b1e","Type":"ContainerStarted","Data":"f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f"} Sep 09 20:54:11.124924 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:11.124414 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dafe8c25-2709-4617-9725-695d19a09b1e","Type":"ContainerStarted","Data":"9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c"} Sep 09 20:54:11.124924 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:11.124453 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dafe8c25-2709-4617-9725-695d19a09b1e","Type":"ContainerStarted","Data":"e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e"} Sep 09 20:54:11.156824 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:11.156758 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.2403689 podStartE2EDuration="11.156735363s" totalImagesPullingTime="7.890795164s" totalInitContainerRuntime="25.571299ms" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:11.154240914 +0000 UTC m=+112.196240715" watchObservedRunningTime="2026-09-09 20:54:11.156735363 +0000 UTC m=+112.198735173" Sep 09 20:54:15.436706 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:15.436665 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:54:16.140828 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:16.140785 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:16.200435 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:16.200396 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:18.523083 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:18.523042 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:54:18.523497 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:18.523127 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:54:18.799408 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:18.799292 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-85c765599c-tlgn6" podUID="2da781ca-4940-4506-ab3f-20fdff5b4efc" containerName="console" containerID="cri-o://315ddea3d4d1bf8b717aaab09705a0b3a42ebd883c0750eb038da23bf31b3ade" gracePeriod=15 Sep 09 20:54:19.034099 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.034059 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:19.080786 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.080711 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c765599c-tlgn6" Sep 09 20:54:19.101402 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.101353 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:19.158721 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.158686 2562 generic.go:356] "Generic (PLEG): container finished" podID="2da781ca-4940-4506-ab3f-20fdff5b4efc" containerID="315ddea3d4d1bf8b717aaab09705a0b3a42ebd883c0750eb038da23bf31b3ade" exitCode=0 Sep 09 20:54:19.158894 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.158731 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c765599c-tlgn6" event={"ID":"2da781ca-4940-4506-ab3f-20fdff5b4efc","Type":"ContainerDied","Data":"315ddea3d4d1bf8b717aaab09705a0b3a42ebd883c0750eb038da23bf31b3ade"} Sep 09 20:54:19.158894 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.158738 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c765599c-tlgn6" Sep 09 20:54:19.158894 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.158778 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c765599c-tlgn6" event={"ID":"2da781ca-4940-4506-ab3f-20fdff5b4efc","Type":"ContainerDied","Data":"9d818d3b1849e510a74a4d0ce65d75b21aef752aebaf6422ab464e870e14003e"} Sep 09 20:54:19.158894 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.158805 2562 scope.go:117] "RemoveContainer" containerID="315ddea3d4d1bf8b717aaab09705a0b3a42ebd883c0750eb038da23bf31b3ade" Sep 09 20:54:19.167281 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.167250 2562 scope.go:117] "RemoveContainer" containerID="315ddea3d4d1bf8b717aaab09705a0b3a42ebd883c0750eb038da23bf31b3ade" Sep 09 20:54:19.167571 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:54:19.167551 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"315ddea3d4d1bf8b717aaab09705a0b3a42ebd883c0750eb038da23bf31b3ade\": container with ID starting with 315ddea3d4d1bf8b717aaab09705a0b3a42ebd883c0750eb038da23bf31b3ade not found: ID does not exist" containerID="315ddea3d4d1bf8b717aaab09705a0b3a42ebd883c0750eb038da23bf31b3ade" Sep 09 20:54:19.167628 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.167580 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"315ddea3d4d1bf8b717aaab09705a0b3a42ebd883c0750eb038da23bf31b3ade"} err="failed to get container status \"315ddea3d4d1bf8b717aaab09705a0b3a42ebd883c0750eb038da23bf31b3ade\": rpc error: code = NotFound desc = could not find container \"315ddea3d4d1bf8b717aaab09705a0b3a42ebd883c0750eb038da23bf31b3ade\": container with ID starting with 315ddea3d4d1bf8b717aaab09705a0b3a42ebd883c0750eb038da23bf31b3ade not found: ID does not exist" Sep 09 20:54:19.181402 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.181371 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2da781ca-4940-4506-ab3f-20fdff5b4efc-console-config\") pod \"2da781ca-4940-4506-ab3f-20fdff5b4efc\" (UID: \"2da781ca-4940-4506-ab3f-20fdff5b4efc\") " Sep 09 20:54:19.181542 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.181418 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2da781ca-4940-4506-ab3f-20fdff5b4efc-oauth-serving-cert\") pod \"2da781ca-4940-4506-ab3f-20fdff5b4efc\" (UID: \"2da781ca-4940-4506-ab3f-20fdff5b4efc\") " Sep 09 20:54:19.181542 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.181446 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-449qj\" (UniqueName: \"kubernetes.io/projected/2da781ca-4940-4506-ab3f-20fdff5b4efc-kube-api-access-449qj\") pod \"2da781ca-4940-4506-ab3f-20fdff5b4efc\" (UID: \"2da781ca-4940-4506-ab3f-20fdff5b4efc\") " Sep 09 20:54:19.181542 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.181492 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2da781ca-4940-4506-ab3f-20fdff5b4efc-service-ca\") pod \"2da781ca-4940-4506-ab3f-20fdff5b4efc\" (UID: \"2da781ca-4940-4506-ab3f-20fdff5b4efc\") " Sep 09 20:54:19.181542 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.181520 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2da781ca-4940-4506-ab3f-20fdff5b4efc-console-serving-cert\") pod \"2da781ca-4940-4506-ab3f-20fdff5b4efc\" (UID: \"2da781ca-4940-4506-ab3f-20fdff5b4efc\") " Sep 09 20:54:19.181732 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.181620 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2da781ca-4940-4506-ab3f-20fdff5b4efc-console-oauth-config\") pod \"2da781ca-4940-4506-ab3f-20fdff5b4efc\" (UID: \"2da781ca-4940-4506-ab3f-20fdff5b4efc\") " Sep 09 20:54:19.181822 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.181792 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da781ca-4940-4506-ab3f-20fdff5b4efc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2da781ca-4940-4506-ab3f-20fdff5b4efc" (UID: "2da781ca-4940-4506-ab3f-20fdff5b4efc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:54:19.181877 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.181799 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da781ca-4940-4506-ab3f-20fdff5b4efc-console-config" (OuterVolumeSpecName: "console-config") pod "2da781ca-4940-4506-ab3f-20fdff5b4efc" (UID: "2da781ca-4940-4506-ab3f-20fdff5b4efc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:54:19.181877 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.181860 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da781ca-4940-4506-ab3f-20fdff5b4efc-service-ca" (OuterVolumeSpecName: "service-ca") pod "2da781ca-4940-4506-ab3f-20fdff5b4efc" (UID: "2da781ca-4940-4506-ab3f-20fdff5b4efc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:54:19.183898 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.183877 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da781ca-4940-4506-ab3f-20fdff5b4efc-kube-api-access-449qj" (OuterVolumeSpecName: "kube-api-access-449qj") pod "2da781ca-4940-4506-ab3f-20fdff5b4efc" (UID: "2da781ca-4940-4506-ab3f-20fdff5b4efc"). InnerVolumeSpecName "kube-api-access-449qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:54:19.184002 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.183910 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2da781ca-4940-4506-ab3f-20fdff5b4efc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2da781ca-4940-4506-ab3f-20fdff5b4efc" (UID: "2da781ca-4940-4506-ab3f-20fdff5b4efc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:54:19.184002 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.183921 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2da781ca-4940-4506-ab3f-20fdff5b4efc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2da781ca-4940-4506-ab3f-20fdff5b4efc" (UID: "2da781ca-4940-4506-ab3f-20fdff5b4efc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:54:19.282610 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.282570 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2da781ca-4940-4506-ab3f-20fdff5b4efc-console-serving-cert\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:54:19.282610 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.282603 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2da781ca-4940-4506-ab3f-20fdff5b4efc-console-oauth-config\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:54:19.282610 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.282614 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2da781ca-4940-4506-ab3f-20fdff5b4efc-console-config\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:54:19.282851 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.282624 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2da781ca-4940-4506-ab3f-20fdff5b4efc-oauth-serving-cert\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:54:19.282851 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.282633 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-449qj\" (UniqueName: \"kubernetes.io/projected/2da781ca-4940-4506-ab3f-20fdff5b4efc-kube-api-access-449qj\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:54:19.282851 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.282643 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2da781ca-4940-4506-ab3f-20fdff5b4efc-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:54:19.479479 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.479448 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85c765599c-tlgn6"] Sep 09 20:54:19.483476 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.483443 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85c765599c-tlgn6"] Sep 09 20:54:19.489401 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:19.489370 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2da781ca-4940-4506-ab3f-20fdff5b4efc" path="/var/lib/kubelet/pods/2da781ca-4940-4506-ab3f-20fdff5b4efc/volumes" Sep 09 20:54:31.142317 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:31.142278 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:31.204865 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:31.204824 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:54:38.528060 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:38.528028 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:54:38.531913 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:38.531887 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-646bcb899d-882pt" Sep 09 20:54:44.721281 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:54:44.721241 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:00.436742 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:00.436702 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:00.452837 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:00.452805 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:01.346196 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:01.346163 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:13.096366 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:13.096268 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:13.857647 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:13.857611 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:18.503297 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.503260 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 20:55:18.503761 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.503563 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="prometheus" containerID="cri-o://1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b" gracePeriod=600 Sep 09 20:55:18.503761 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.503574 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="kube-rbac-proxy" containerID="cri-o://f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f" gracePeriod=600 Sep 09 20:55:18.503761 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.503594 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="thanos-sidecar" containerID="cri-o://e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e" gracePeriod=600 Sep 09 20:55:18.503761 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.503612 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="config-reloader" containerID="cri-o://0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b" gracePeriod=600 Sep 09 20:55:18.503761 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.503647 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="kube-rbac-proxy-web" containerID="cri-o://9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c" gracePeriod=600 Sep 09 20:55:18.503761 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.503736 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="kube-rbac-proxy-thanos" containerID="cri-o://9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880" gracePeriod=600 Sep 09 20:55:18.759870 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.759795 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:18.921046 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.921008 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-grpc-tls\") pod \"dafe8c25-2709-4617-9725-695d19a09b1e\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " Sep 09 20:55:18.921046 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.921052 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-configmap-metrics-client-ca\") pod \"dafe8c25-2709-4617-9725-695d19a09b1e\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " Sep 09 20:55:18.921301 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.921077 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-metrics-client-certs\") pod \"dafe8c25-2709-4617-9725-695d19a09b1e\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " Sep 09 20:55:18.921301 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.921121 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-web-config\") pod \"dafe8c25-2709-4617-9725-695d19a09b1e\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " Sep 09 20:55:18.921301 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.921150 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-kube-rbac-proxy\") pod \"dafe8c25-2709-4617-9725-695d19a09b1e\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " Sep 09 20:55:18.921301 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.921176 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8v4zt\" (UniqueName: \"kubernetes.io/projected/dafe8c25-2709-4617-9725-695d19a09b1e-kube-api-access-8v4zt\") pod \"dafe8c25-2709-4617-9725-695d19a09b1e\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " Sep 09 20:55:18.921301 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.921202 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-thanos-prometheus-http-client-file\") pod \"dafe8c25-2709-4617-9725-695d19a09b1e\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " Sep 09 20:55:18.921301 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.921237 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dafe8c25-2709-4617-9725-695d19a09b1e-tls-assets\") pod \"dafe8c25-2709-4617-9725-695d19a09b1e\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " Sep 09 20:55:18.921301 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.921263 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"dafe8c25-2709-4617-9725-695d19a09b1e\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " Sep 09 20:55:18.921301 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.921286 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-configmap-serving-certs-ca-bundle\") pod \"dafe8c25-2709-4617-9725-695d19a09b1e\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " Sep 09 20:55:18.921813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.921318 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-trusted-ca-bundle\") pod \"dafe8c25-2709-4617-9725-695d19a09b1e\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " Sep 09 20:55:18.921813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.921347 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-config\") pod \"dafe8c25-2709-4617-9725-695d19a09b1e\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " Sep 09 20:55:18.921813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.921380 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"dafe8c25-2709-4617-9725-695d19a09b1e\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " Sep 09 20:55:18.921813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.921411 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-k8s-rulefiles-0\") pod \"dafe8c25-2709-4617-9725-695d19a09b1e\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " Sep 09 20:55:18.921813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.921466 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-k8s-rulefiles-1\") pod \"dafe8c25-2709-4617-9725-695d19a09b1e\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " Sep 09 20:55:18.921813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.921463 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "dafe8c25-2709-4617-9725-695d19a09b1e" (UID: "dafe8c25-2709-4617-9725-695d19a09b1e"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:55:18.921813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.921498 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-configmap-kubelet-serving-ca-bundle\") pod \"dafe8c25-2709-4617-9725-695d19a09b1e\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " Sep 09 20:55:18.921813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.921548 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dafe8c25-2709-4617-9725-695d19a09b1e-config-out\") pod \"dafe8c25-2709-4617-9725-695d19a09b1e\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " Sep 09 20:55:18.921813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.921573 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-prometheus-k8s-tls\") pod \"dafe8c25-2709-4617-9725-695d19a09b1e\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " Sep 09 20:55:18.921813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.921596 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-k8s-db\") pod \"dafe8c25-2709-4617-9725-695d19a09b1e\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " Sep 09 20:55:18.921813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.921632 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-k8s-rulefiles-2\") pod \"dafe8c25-2709-4617-9725-695d19a09b1e\" (UID: \"dafe8c25-2709-4617-9725-695d19a09b1e\") " Sep 09 20:55:18.922398 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.921814 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "dafe8c25-2709-4617-9725-695d19a09b1e" (UID: "dafe8c25-2709-4617-9725-695d19a09b1e"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:55:18.922398 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.921906 2562 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-configmap-metrics-client-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:55:18.922398 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.921925 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:55:18.922398 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.922156 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "dafe8c25-2709-4617-9725-695d19a09b1e" (UID: "dafe8c25-2709-4617-9725-695d19a09b1e"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:55:18.923062 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.923028 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "dafe8c25-2709-4617-9725-695d19a09b1e" (UID: "dafe8c25-2709-4617-9725-695d19a09b1e"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:55:18.923209 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.923180 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "dafe8c25-2709-4617-9725-695d19a09b1e" (UID: "dafe8c25-2709-4617-9725-695d19a09b1e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:55:18.923468 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.923445 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "dafe8c25-2709-4617-9725-695d19a09b1e" (UID: "dafe8c25-2709-4617-9725-695d19a09b1e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:55:18.923736 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.923705 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "dafe8c25-2709-4617-9725-695d19a09b1e" (UID: "dafe8c25-2709-4617-9725-695d19a09b1e"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:55:18.924415 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.924386 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "dafe8c25-2709-4617-9725-695d19a09b1e" (UID: "dafe8c25-2709-4617-9725-695d19a09b1e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 20:55:18.924618 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.924593 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "dafe8c25-2709-4617-9725-695d19a09b1e" (UID: "dafe8c25-2709-4617-9725-695d19a09b1e"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:55:18.925026 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.924990 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "dafe8c25-2709-4617-9725-695d19a09b1e" (UID: "dafe8c25-2709-4617-9725-695d19a09b1e"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:55:18.925951 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.925803 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "dafe8c25-2709-4617-9725-695d19a09b1e" (UID: "dafe8c25-2709-4617-9725-695d19a09b1e"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:55:18.926070 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.925977 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "dafe8c25-2709-4617-9725-695d19a09b1e" (UID: "dafe8c25-2709-4617-9725-695d19a09b1e"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:55:18.926070 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.926058 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafe8c25-2709-4617-9725-695d19a09b1e-config-out" (OuterVolumeSpecName: "config-out") pod "dafe8c25-2709-4617-9725-695d19a09b1e" (UID: "dafe8c25-2709-4617-9725-695d19a09b1e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 20:55:18.926335 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.926312 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-config" (OuterVolumeSpecName: "config") pod "dafe8c25-2709-4617-9725-695d19a09b1e" (UID: "dafe8c25-2709-4617-9725-695d19a09b1e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:55:18.926614 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.926571 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafe8c25-2709-4617-9725-695d19a09b1e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "dafe8c25-2709-4617-9725-695d19a09b1e" (UID: "dafe8c25-2709-4617-9725-695d19a09b1e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:55:18.926713 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.926649 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "dafe8c25-2709-4617-9725-695d19a09b1e" (UID: "dafe8c25-2709-4617-9725-695d19a09b1e"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:55:18.926713 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.926676 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafe8c25-2709-4617-9725-695d19a09b1e-kube-api-access-8v4zt" (OuterVolumeSpecName: "kube-api-access-8v4zt") pod "dafe8c25-2709-4617-9725-695d19a09b1e" (UID: "dafe8c25-2709-4617-9725-695d19a09b1e"). InnerVolumeSpecName "kube-api-access-8v4zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 20:55:18.926798 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.926759 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "dafe8c25-2709-4617-9725-695d19a09b1e" (UID: "dafe8c25-2709-4617-9725-695d19a09b1e"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:55:18.926956 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.926915 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "dafe8c25-2709-4617-9725-695d19a09b1e" (UID: "dafe8c25-2709-4617-9725-695d19a09b1e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:55:18.936434 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:18.936410 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-web-config" (OuterVolumeSpecName: "web-config") pod "dafe8c25-2709-4617-9725-695d19a09b1e" (UID: "dafe8c25-2709-4617-9725-695d19a09b1e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 20:55:19.023145 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.023039 2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-web-config\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:55:19.023145 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.023082 2562 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-kube-rbac-proxy\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:55:19.023145 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.023092 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v4zt\" (UniqueName: \"kubernetes.io/projected/dafe8c25-2709-4617-9725-695d19a09b1e-kube-api-access-8v4zt\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:55:19.023145 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.023103 2562 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:55:19.023145 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.023113 2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dafe8c25-2709-4617-9725-695d19a09b1e-tls-assets\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:55:19.023145 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.023122 2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:55:19.023145 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.023130 2562 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:55:19.023145 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.023140 2562 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-config\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:55:19.023145 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.023149 2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:55:19.023145 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.023158 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:55:19.023145 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.023167 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:55:19.023647 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.023177 2562 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:55:19.023647 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.023186 2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dafe8c25-2709-4617-9725-695d19a09b1e-config-out\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:55:19.023647 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.023195 2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:55:19.023647 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.023214 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-k8s-db\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:55:19.023647 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.023221 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dafe8c25-2709-4617-9725-695d19a09b1e-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:55:19.023647 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.023230 2562 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-grpc-tls\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:55:19.023647 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.023237 2562 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dafe8c25-2709-4617-9725-695d19a09b1e-secret-metrics-client-certs\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 20:55:19.404114 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.404077 2562 generic.go:356] "Generic (PLEG): container finished" podID="dafe8c25-2709-4617-9725-695d19a09b1e" containerID="9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880" exitCode=0 Sep 09 20:55:19.404114 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.404106 2562 generic.go:356] "Generic (PLEG): container finished" podID="dafe8c25-2709-4617-9725-695d19a09b1e" containerID="f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f" exitCode=0 Sep 09 20:55:19.404114 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.404116 2562 generic.go:356] "Generic (PLEG): container finished" podID="dafe8c25-2709-4617-9725-695d19a09b1e" containerID="9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c" exitCode=0 Sep 09 20:55:19.404114 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.404125 2562 generic.go:356] "Generic (PLEG): container finished" podID="dafe8c25-2709-4617-9725-695d19a09b1e" containerID="e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e" exitCode=0 Sep 09 20:55:19.404442 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.404133 2562 generic.go:356] "Generic (PLEG): container finished" podID="dafe8c25-2709-4617-9725-695d19a09b1e" containerID="0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b" exitCode=0 Sep 09 20:55:19.404442 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.404141 2562 generic.go:356] "Generic (PLEG): container finished" podID="dafe8c25-2709-4617-9725-695d19a09b1e" containerID="1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b" exitCode=0 Sep 09 20:55:19.404442 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.404151 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.404442 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.404157 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dafe8c25-2709-4617-9725-695d19a09b1e","Type":"ContainerDied","Data":"9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880"} Sep 09 20:55:19.404442 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.404226 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dafe8c25-2709-4617-9725-695d19a09b1e","Type":"ContainerDied","Data":"f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f"} Sep 09 20:55:19.404442 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.404253 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dafe8c25-2709-4617-9725-695d19a09b1e","Type":"ContainerDied","Data":"9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c"} Sep 09 20:55:19.404442 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.404278 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dafe8c25-2709-4617-9725-695d19a09b1e","Type":"ContainerDied","Data":"e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e"} Sep 09 20:55:19.404442 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.404301 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dafe8c25-2709-4617-9725-695d19a09b1e","Type":"ContainerDied","Data":"0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b"} Sep 09 20:55:19.404442 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.404301 2562 scope.go:117] "RemoveContainer" containerID="9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880" Sep 09 20:55:19.404442 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.404324 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dafe8c25-2709-4617-9725-695d19a09b1e","Type":"ContainerDied","Data":"1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b"} Sep 09 20:55:19.404442 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.404363 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dafe8c25-2709-4617-9725-695d19a09b1e","Type":"ContainerDied","Data":"0f80c9e2de8c1df0e3986ab87650db6096df021fa2fa17cffc0820b0a3659b9a"} Sep 09 20:55:19.412384 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.412366 2562 scope.go:117] "RemoveContainer" containerID="f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f" Sep 09 20:55:19.419441 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.419424 2562 scope.go:117] "RemoveContainer" containerID="9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c" Sep 09 20:55:19.426101 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.426085 2562 scope.go:117] "RemoveContainer" containerID="e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e" Sep 09 20:55:19.434191 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.434159 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 20:55:19.436447 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.436418 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 20:55:19.438213 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.438182 2562 scope.go:117] "RemoveContainer" containerID="0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b" Sep 09 20:55:19.445450 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.445427 2562 scope.go:117] "RemoveContainer" containerID="1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b" Sep 09 20:55:19.453092 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.453068 2562 scope.go:117] "RemoveContainer" containerID="779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed" Sep 09 20:55:19.460695 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.460680 2562 scope.go:117] "RemoveContainer" containerID="9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880" Sep 09 20:55:19.461358 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.461337 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 20:55:19.461674 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.461660 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="init-config-reloader" Sep 09 20:55:19.461714 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.461676 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="init-config-reloader" Sep 09 20:55:19.461714 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.461685 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="thanos-sidecar" Sep 09 20:55:19.461714 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.461691 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="thanos-sidecar" Sep 09 20:55:19.461714 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.461701 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="kube-rbac-proxy" Sep 09 20:55:19.461714 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.461710 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="kube-rbac-proxy" Sep 09 20:55:19.461860 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.461720 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="kube-rbac-proxy-thanos" Sep 09 20:55:19.461860 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.461726 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="kube-rbac-proxy-thanos" Sep 09 20:55:19.461860 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.461733 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="config-reloader" Sep 09 20:55:19.461860 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.461739 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="config-reloader" Sep 09 20:55:19.461860 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.461748 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2da781ca-4940-4506-ab3f-20fdff5b4efc" containerName="console" Sep 09 20:55:19.461860 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.461753 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2da781ca-4940-4506-ab3f-20fdff5b4efc" containerName="console" Sep 09 20:55:19.461860 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.461760 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="prometheus" Sep 09 20:55:19.461860 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.461765 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="prometheus" Sep 09 20:55:19.461860 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.461770 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="kube-rbac-proxy-web" Sep 09 20:55:19.461860 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.461775 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="kube-rbac-proxy-web" Sep 09 20:55:19.461860 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.461821 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2da781ca-4940-4506-ab3f-20fdff5b4efc" containerName="console" Sep 09 20:55:19.461860 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.461831 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="kube-rbac-proxy" Sep 09 20:55:19.461860 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.461838 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="config-reloader" Sep 09 20:55:19.461860 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.461844 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="prometheus" Sep 09 20:55:19.461860 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.461850 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="kube-rbac-proxy-web" Sep 09 20:55:19.461860 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.461855 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="thanos-sidecar" Sep 09 20:55:19.461860 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.461861 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" containerName="kube-rbac-proxy-thanos" Sep 09 20:55:19.462597 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:55:19.462164 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880\": container with ID starting with 9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880 not found: ID does not exist" containerID="9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880" Sep 09 20:55:19.462597 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.462197 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880"} err="failed to get container status \"9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880\": rpc error: code = NotFound desc = could not find container \"9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880\": container with ID starting with 9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880 not found: ID does not exist" Sep 09 20:55:19.462597 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.462223 2562 scope.go:117] "RemoveContainer" containerID="f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f" Sep 09 20:55:19.463000 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:55:19.462972 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f\": container with ID starting with f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f not found: ID does not exist" containerID="f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f" Sep 09 20:55:19.463093 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.463010 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f"} err="failed to get container status \"f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f\": rpc error: code = NotFound desc = could not find container \"f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f\": container with ID starting with f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f not found: ID does not exist" Sep 09 20:55:19.463093 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.463035 2562 scope.go:117] "RemoveContainer" containerID="9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c" Sep 09 20:55:19.463369 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:55:19.463346 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c\": container with ID starting with 9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c not found: ID does not exist" containerID="9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c" Sep 09 20:55:19.463544 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.463377 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c"} err="failed to get container status \"9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c\": rpc error: code = NotFound desc = could not find container \"9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c\": container with ID starting with 9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c not found: ID does not exist" Sep 09 20:55:19.463544 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.463398 2562 scope.go:117] "RemoveContainer" containerID="e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e" Sep 09 20:55:19.465777 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:55:19.463854 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e\": container with ID starting with e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e not found: ID does not exist" containerID="e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e" Sep 09 20:55:19.465777 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.463882 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e"} err="failed to get container status \"e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e\": rpc error: code = NotFound desc = could not find container \"e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e\": container with ID starting with e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e not found: ID does not exist" Sep 09 20:55:19.465777 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.463919 2562 scope.go:117] "RemoveContainer" containerID="0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b" Sep 09 20:55:19.465777 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:55:19.464425 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b\": container with ID starting with 0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b not found: ID does not exist" containerID="0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b" Sep 09 20:55:19.465777 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.464450 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b"} err="failed to get container status \"0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b\": rpc error: code = NotFound desc = could not find container \"0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b\": container with ID starting with 0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b not found: ID does not exist" Sep 09 20:55:19.465777 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.464470 2562 scope.go:117] "RemoveContainer" containerID="1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b" Sep 09 20:55:19.465777 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:55:19.464980 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b\": container with ID starting with 1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b not found: ID does not exist" containerID="1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b" Sep 09 20:55:19.465777 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.465004 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b"} err="failed to get container status \"1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b\": rpc error: code = NotFound desc = could not find container \"1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b\": container with ID starting with 1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b not found: ID does not exist" Sep 09 20:55:19.465777 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.465021 2562 scope.go:117] "RemoveContainer" containerID="779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed" Sep 09 20:55:19.465777 ip-10-0-136-126 kubenswrapper[2562]: E0909 20:55:19.465463 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed\": container with ID starting with 779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed not found: ID does not exist" containerID="779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed" Sep 09 20:55:19.465777 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.465488 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed"} err="failed to get container status \"779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed\": rpc error: code = NotFound desc = could not find container \"779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed\": container with ID starting with 779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed not found: ID does not exist" Sep 09 20:55:19.465777 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.465505 2562 scope.go:117] "RemoveContainer" containerID="9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880" Sep 09 20:55:19.466482 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.465984 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880"} err="failed to get container status \"9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880\": rpc error: code = NotFound desc = could not find container \"9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880\": container with ID starting with 9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880 not found: ID does not exist" Sep 09 20:55:19.466482 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.466007 2562 scope.go:117] "RemoveContainer" containerID="f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f" Sep 09 20:55:19.466482 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.466286 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f"} err="failed to get container status \"f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f\": rpc error: code = NotFound desc = could not find container \"f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f\": container with ID starting with f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f not found: ID does not exist" Sep 09 20:55:19.466482 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.466309 2562 scope.go:117] "RemoveContainer" containerID="9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c" Sep 09 20:55:19.466688 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.466540 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c"} err="failed to get container status \"9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c\": rpc error: code = NotFound desc = could not find container \"9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c\": container with ID starting with 9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c not found: ID does not exist" Sep 09 20:55:19.466688 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.466561 2562 scope.go:117] "RemoveContainer" containerID="e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e" Sep 09 20:55:19.466796 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.466778 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e"} err="failed to get container status \"e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e\": rpc error: code = NotFound desc = could not find container \"e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e\": container with ID starting with e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e not found: ID does not exist" Sep 09 20:55:19.466862 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.466797 2562 scope.go:117] "RemoveContainer" containerID="0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b" Sep 09 20:55:19.467026 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.467004 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b"} err="failed to get container status \"0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b\": rpc error: code = NotFound desc = could not find container \"0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b\": container with ID starting with 0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b not found: ID does not exist" Sep 09 20:55:19.467094 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.467026 2562 scope.go:117] "RemoveContainer" containerID="1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b" Sep 09 20:55:19.467261 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.467238 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b"} err="failed to get container status \"1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b\": rpc error: code = NotFound desc = could not find container \"1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b\": container with ID starting with 1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b not found: ID does not exist" Sep 09 20:55:19.467306 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.467263 2562 scope.go:117] "RemoveContainer" containerID="779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed" Sep 09 20:55:19.467539 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.467518 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed"} err="failed to get container status \"779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed\": rpc error: code = NotFound desc = could not find container \"779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed\": container with ID starting with 779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed not found: ID does not exist" Sep 09 20:55:19.467592 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.467541 2562 scope.go:117] "RemoveContainer" containerID="9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880" Sep 09 20:55:19.467820 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.467798 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880"} err="failed to get container status \"9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880\": rpc error: code = NotFound desc = could not find container \"9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880\": container with ID starting with 9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880 not found: ID does not exist" Sep 09 20:55:19.467820 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.467821 2562 scope.go:117] "RemoveContainer" containerID="f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f" Sep 09 20:55:19.467970 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.467915 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.468508 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.468439 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f"} err="failed to get container status \"f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f\": rpc error: code = NotFound desc = could not find container \"f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f\": container with ID starting with f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f not found: ID does not exist" Sep 09 20:55:19.468606 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.468514 2562 scope.go:117] "RemoveContainer" containerID="9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c" Sep 09 20:55:19.469200 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.469166 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c"} err="failed to get container status \"9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c\": rpc error: code = NotFound desc = could not find container \"9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c\": container with ID starting with 9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c not found: ID does not exist" Sep 09 20:55:19.469295 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.469202 2562 scope.go:117] "RemoveContainer" containerID="e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e" Sep 09 20:55:19.469469 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.469441 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e"} err="failed to get container status \"e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e\": rpc error: code = NotFound desc = could not find container \"e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e\": container with ID starting with e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e not found: ID does not exist" Sep 09 20:55:19.469469 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.469467 2562 scope.go:117] "RemoveContainer" containerID="0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b" Sep 09 20:55:19.469722 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.469704 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b"} err="failed to get container status \"0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b\": rpc error: code = NotFound desc = could not find container \"0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b\": container with ID starting with 0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b not found: ID does not exist" Sep 09 20:55:19.469805 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.469724 2562 scope.go:117] "RemoveContainer" containerID="1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b" Sep 09 20:55:19.469987 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.469968 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b"} err="failed to get container status \"1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b\": rpc error: code = NotFound desc = could not find container \"1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b\": container with ID starting with 1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b not found: ID does not exist" Sep 09 20:55:19.470072 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.469988 2562 scope.go:117] "RemoveContainer" containerID="779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed" Sep 09 20:55:19.470202 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.470185 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed"} err="failed to get container status \"779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed\": rpc error: code = NotFound desc = could not find container \"779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed\": container with ID starting with 779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed not found: ID does not exist" Sep 09 20:55:19.470264 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.470203 2562 scope.go:117] "RemoveContainer" containerID="9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880" Sep 09 20:55:19.470435 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.470408 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880"} err="failed to get container status \"9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880\": rpc error: code = NotFound desc = could not find container \"9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880\": container with ID starting with 9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880 not found: ID does not exist" Sep 09 20:55:19.470435 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.470434 2562 scope.go:117] "RemoveContainer" containerID="f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f" Sep 09 20:55:19.470674 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.470651 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 09 20:55:19.470779 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.470667 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 09 20:55:19.470779 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.470755 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 09 20:55:19.470779 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.470757 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f"} err="failed to get container status \"f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f\": rpc error: code = NotFound desc = could not find container \"f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f\": container with ID starting with f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f not found: ID does not exist" Sep 09 20:55:19.470965 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.470783 2562 scope.go:117] "RemoveContainer" containerID="9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c" Sep 09 20:55:19.470965 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.470848 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 09 20:55:19.471066 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.471011 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 09 20:55:19.471066 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.471042 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c"} err="failed to get container status \"9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c\": rpc error: code = NotFound desc = could not find container \"9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c\": container with ID starting with 9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c not found: ID does not exist" Sep 09 20:55:19.471167 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.471066 2562 scope.go:117] "RemoveContainer" containerID="e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e" Sep 09 20:55:19.471167 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.471081 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 09 20:55:19.471275 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.471260 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qjsfk\"" Sep 09 20:55:19.471477 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.471340 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 09 20:55:19.471477 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.471362 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 09 20:55:19.471477 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.471409 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6c83evuihm204\"" Sep 09 20:55:19.471477 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.471431 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 09 20:55:19.471477 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.471360 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e"} err="failed to get container status \"e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e\": rpc error: code = NotFound desc = could not find container \"e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e\": container with ID starting with e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e not found: ID does not exist" Sep 09 20:55:19.471477 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.471455 2562 scope.go:117] "RemoveContainer" containerID="0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b" Sep 09 20:55:19.471798 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.471531 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 09 20:55:19.471798 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.471599 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 09 20:55:19.472069 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.472038 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b"} err="failed to get container status \"0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b\": rpc error: code = NotFound desc = could not find container \"0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b\": container with ID starting with 0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b not found: ID does not exist" Sep 09 20:55:19.472069 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.472067 2562 scope.go:117] "RemoveContainer" containerID="1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b" Sep 09 20:55:19.472239 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.472076 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 09 20:55:19.472587 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.472486 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b"} err="failed to get container status \"1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b\": rpc error: code = NotFound desc = could not find container \"1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b\": container with ID starting with 1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b not found: ID does not exist" Sep 09 20:55:19.472587 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.472514 2562 scope.go:117] "RemoveContainer" containerID="779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed" Sep 09 20:55:19.472811 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.472788 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed"} err="failed to get container status \"779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed\": rpc error: code = NotFound desc = could not find container \"779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed\": container with ID starting with 779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed not found: ID does not exist" Sep 09 20:55:19.472884 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.472812 2562 scope.go:117] "RemoveContainer" containerID="9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880" Sep 09 20:55:19.473204 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.473172 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880"} err="failed to get container status \"9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880\": rpc error: code = NotFound desc = could not find container \"9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880\": container with ID starting with 9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880 not found: ID does not exist" Sep 09 20:55:19.473204 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.473201 2562 scope.go:117] "RemoveContainer" containerID="f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f" Sep 09 20:55:19.473669 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.473649 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f"} err="failed to get container status \"f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f\": rpc error: code = NotFound desc = could not find container \"f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f\": container with ID starting with f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f not found: ID does not exist" Sep 09 20:55:19.473669 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.473669 2562 scope.go:117] "RemoveContainer" containerID="9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c" Sep 09 20:55:19.474050 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.474019 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c"} err="failed to get container status \"9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c\": rpc error: code = NotFound desc = could not find container \"9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c\": container with ID starting with 9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c not found: ID does not exist" Sep 09 20:55:19.474166 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.474052 2562 scope.go:117] "RemoveContainer" containerID="e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e" Sep 09 20:55:19.474319 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.474288 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e"} err="failed to get container status \"e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e\": rpc error: code = NotFound desc = could not find container \"e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e\": container with ID starting with e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e not found: ID does not exist" Sep 09 20:55:19.474319 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.474315 2562 scope.go:117] "RemoveContainer" containerID="0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b" Sep 09 20:55:19.474427 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.474414 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 09 20:55:19.474548 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.474525 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b"} err="failed to get container status \"0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b\": rpc error: code = NotFound desc = could not find container \"0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b\": container with ID starting with 0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b not found: ID does not exist" Sep 09 20:55:19.474749 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.474550 2562 scope.go:117] "RemoveContainer" containerID="1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b" Sep 09 20:55:19.474836 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.474815 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b"} err="failed to get container status \"1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b\": rpc error: code = NotFound desc = could not find container \"1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b\": container with ID starting with 1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b not found: ID does not exist" Sep 09 20:55:19.474836 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.474833 2562 scope.go:117] "RemoveContainer" containerID="779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed" Sep 09 20:55:19.475151 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.475127 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed"} err="failed to get container status \"779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed\": rpc error: code = NotFound desc = could not find container \"779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed\": container with ID starting with 779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed not found: ID does not exist" Sep 09 20:55:19.475241 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.475152 2562 scope.go:117] "RemoveContainer" containerID="9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880" Sep 09 20:55:19.475615 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.475595 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880"} err="failed to get container status \"9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880\": rpc error: code = NotFound desc = could not find container \"9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880\": container with ID starting with 9dbf6c4f61a063766ec4985d3e838f362b0e5aaf0c0dd45c267ff06a4ead5880 not found: ID does not exist" Sep 09 20:55:19.475717 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.475630 2562 scope.go:117] "RemoveContainer" containerID="f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f" Sep 09 20:55:19.476025 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.475954 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f"} err="failed to get container status \"f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f\": rpc error: code = NotFound desc = could not find container \"f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f\": container with ID starting with f968cb65e9f6bde23cb12a154dbfc6bd66671e41e112e43b05d60e542111777f not found: ID does not exist" Sep 09 20:55:19.476025 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.475981 2562 scope.go:117] "RemoveContainer" containerID="9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c" Sep 09 20:55:19.476274 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.476251 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c"} err="failed to get container status \"9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c\": rpc error: code = NotFound desc = could not find container \"9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c\": container with ID starting with 9c25a6c7c337c4d2fa9ce8d43205de9852c3cc95386a303c7a9c0f2609031c9c not found: ID does not exist" Sep 09 20:55:19.476344 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.476277 2562 scope.go:117] "RemoveContainer" containerID="e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e" Sep 09 20:55:19.476585 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.476545 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e"} err="failed to get container status \"e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e\": rpc error: code = NotFound desc = could not find container \"e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e\": container with ID starting with e975491c57b35a81b686ee243c2bc18a54087e3b90d7ebcd8f3d65b6a3149b7e not found: ID does not exist" Sep 09 20:55:19.476585 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.476568 2562 scope.go:117] "RemoveContainer" containerID="0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b" Sep 09 20:55:19.476868 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.476555 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 09 20:55:19.477031 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.476997 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b"} err="failed to get container status \"0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b\": rpc error: code = NotFound desc = could not find container \"0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b\": container with ID starting with 0b332a97f95495098b059890d6916312d01f7110d25a0d4ce950291a3b94808b not found: ID does not exist" Sep 09 20:55:19.477031 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.477027 2562 scope.go:117] "RemoveContainer" containerID="1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b" Sep 09 20:55:19.477313 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.477294 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b"} err="failed to get container status \"1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b\": rpc error: code = NotFound desc = could not find container \"1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b\": container with ID starting with 1220249d09861adb59f84d722ace7368620bea35de92f11ef605e7277526c66b not found: ID does not exist" Sep 09 20:55:19.477370 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.477314 2562 scope.go:117] "RemoveContainer" containerID="779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed" Sep 09 20:55:19.477542 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.477523 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed"} err="failed to get container status \"779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed\": rpc error: code = NotFound desc = could not find container \"779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed\": container with ID starting with 779e18be8e715ea5b1bd7bcb2f9914043109b21cfdc38b149930a29bb80658ed not found: ID does not exist" Sep 09 20:55:19.490519 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.490488 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dafe8c25-2709-4617-9725-695d19a09b1e" path="/var/lib/kubelet/pods/dafe8c25-2709-4617-9725-695d19a09b1e/volumes" Sep 09 20:55:19.491034 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.491019 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 20:55:19.629580 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.629540 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-config\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.629580 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.629579 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b192551-2069-4ef6-a077-f6f22c73d5a3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.630127 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.629601 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3b192551-2069-4ef6-a077-f6f22c73d5a3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.630127 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.629616 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-web-config\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.630127 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.629675 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b192551-2069-4ef6-a077-f6f22c73d5a3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.630127 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.629729 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.630127 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.629767 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3b192551-2069-4ef6-a077-f6f22c73d5a3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.630127 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.629791 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3b192551-2069-4ef6-a077-f6f22c73d5a3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.630127 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.629820 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b192551-2069-4ef6-a077-f6f22c73d5a3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.630127 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.629898 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.630127 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.629925 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3b192551-2069-4ef6-a077-f6f22c73d5a3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.630127 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.629980 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.630127 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.630008 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b192551-2069-4ef6-a077-f6f22c73d5a3-config-out\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.630127 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.630033 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b192551-2069-4ef6-a077-f6f22c73d5a3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.630127 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.630060 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.630127 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.630079 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.630127 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.630102 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.630127 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.630118 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.630617 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.630136 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b192551-2069-4ef6-a077-f6f22c73d5a3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.630617 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.630159 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-727zx\" (UniqueName: \"kubernetes.io/projected/3b192551-2069-4ef6-a077-f6f22c73d5a3-kube-api-access-727zx\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.731300 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.731202 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.731300 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.731239 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.731300 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.731261 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b192551-2069-4ef6-a077-f6f22c73d5a3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.731300 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.731284 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-727zx\" (UniqueName: \"kubernetes.io/projected/3b192551-2069-4ef6-a077-f6f22c73d5a3-kube-api-access-727zx\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.731813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.731307 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-config\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.731813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.731329 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b192551-2069-4ef6-a077-f6f22c73d5a3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.731813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.731355 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3b192551-2069-4ef6-a077-f6f22c73d5a3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.731813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.731378 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-web-config\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.731813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.731405 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b192551-2069-4ef6-a077-f6f22c73d5a3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.731813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.731428 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.731813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.731457 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3b192551-2069-4ef6-a077-f6f22c73d5a3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.731813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.731479 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3b192551-2069-4ef6-a077-f6f22c73d5a3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.731813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.731502 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b192551-2069-4ef6-a077-f6f22c73d5a3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.731813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.731557 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.731813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.731579 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3b192551-2069-4ef6-a077-f6f22c73d5a3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.731813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.731607 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.731813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.731642 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b192551-2069-4ef6-a077-f6f22c73d5a3-config-out\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.731813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.731669 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b192551-2069-4ef6-a077-f6f22c73d5a3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.731813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.731703 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.731813 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.731735 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.735569 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.732006 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3b192551-2069-4ef6-a077-f6f22c73d5a3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.735569 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.732078 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b192551-2069-4ef6-a077-f6f22c73d5a3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.735569 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.732596 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b192551-2069-4ef6-a077-f6f22c73d5a3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.735569 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.732692 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b192551-2069-4ef6-a077-f6f22c73d5a3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.735569 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.733031 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3b192551-2069-4ef6-a077-f6f22c73d5a3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.735569 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.733289 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b192551-2069-4ef6-a077-f6f22c73d5a3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.735569 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.733306 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3b192551-2069-4ef6-a077-f6f22c73d5a3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.735569 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.735248 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.735569 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.735268 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.736070 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.735819 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.736250 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.736227 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.736892 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.736290 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.736892 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.736650 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-config\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.736892 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.736696 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3b192551-2069-4ef6-a077-f6f22c73d5a3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.736892 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.736725 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-web-config\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.736892 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.736803 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.737318 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.737298 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3b192551-2069-4ef6-a077-f6f22c73d5a3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.737549 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.737522 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b192551-2069-4ef6-a077-f6f22c73d5a3-config-out\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.737549 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.737540 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b192551-2069-4ef6-a077-f6f22c73d5a3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.739614 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.739597 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-727zx\" (UniqueName: \"kubernetes.io/projected/3b192551-2069-4ef6-a077-f6f22c73d5a3-kube-api-access-727zx\") pod \"prometheus-k8s-0\" (UID: \"3b192551-2069-4ef6-a077-f6f22c73d5a3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:19.779701 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:19.779659 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:20.062736 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:20.062706 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 20:55:20.063882 ip-10-0-136-126 kubenswrapper[2562]: W0909 20:55:20.063854 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b192551_2069_4ef6_a077_f6f22c73d5a3.slice/crio-9253e57fcddead2f6c8e34af68830849c9c44babec327834089875de9651b8c9 WatchSource:0}: Error finding container 9253e57fcddead2f6c8e34af68830849c9c44babec327834089875de9651b8c9: Status 404 returned error can't find the container with id 9253e57fcddead2f6c8e34af68830849c9c44babec327834089875de9651b8c9 Sep 09 20:55:20.411099 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:20.411063 2562 generic.go:356] "Generic (PLEG): container finished" podID="3b192551-2069-4ef6-a077-f6f22c73d5a3" containerID="4190aa6c036cb0cca0d979825a38dd01046f7952bba31256939f6584f1110a7e" exitCode=0 Sep 09 20:55:20.411283 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:20.411151 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b192551-2069-4ef6-a077-f6f22c73d5a3","Type":"ContainerDied","Data":"4190aa6c036cb0cca0d979825a38dd01046f7952bba31256939f6584f1110a7e"} Sep 09 20:55:20.411283 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:20.411224 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b192551-2069-4ef6-a077-f6f22c73d5a3","Type":"ContainerStarted","Data":"9253e57fcddead2f6c8e34af68830849c9c44babec327834089875de9651b8c9"} Sep 09 20:55:21.241139 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:21.241100 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:21.317370 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:21.317334 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:21.421321 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:21.421279 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b192551-2069-4ef6-a077-f6f22c73d5a3","Type":"ContainerStarted","Data":"7c6d2b433b419bbb2035498abd3aaaddc593ddcdb14d818579564070a8035c80"} Sep 09 20:55:21.421494 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:21.421372 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b192551-2069-4ef6-a077-f6f22c73d5a3","Type":"ContainerStarted","Data":"0cdb7516c14bd99c424c2683dd04edefd7677d7934c913a4db7424b0f2362169"} Sep 09 20:55:21.421494 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:21.421413 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b192551-2069-4ef6-a077-f6f22c73d5a3","Type":"ContainerStarted","Data":"bd610a53c6459553e1f45514bea55461317560a4d86ff6a4bd3b85d120eeb1c4"} Sep 09 20:55:21.421494 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:21.421450 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b192551-2069-4ef6-a077-f6f22c73d5a3","Type":"ContainerStarted","Data":"ebb00479146b3782fbb6e17d2b1375c7016ae531130b61607c51e2bb46b663fa"} Sep 09 20:55:21.421494 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:21.421488 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b192551-2069-4ef6-a077-f6f22c73d5a3","Type":"ContainerStarted","Data":"206794e68351613753985b29627cdfa9659e318ee4f0879fd3eb912c77b1bb41"} Sep 09 20:55:21.421703 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:21.421522 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b192551-2069-4ef6-a077-f6f22c73d5a3","Type":"ContainerStarted","Data":"cd7408e47d7db486a3db79d40ba38fced685b64dc386eee8a27f496de3066620"} Sep 09 20:55:21.449344 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:21.449269 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.427858124 podStartE2EDuration="2.449247794s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.38967ms" isStatefulPod=false podCreationTimestamp="2026-09-09 20:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:21.448576123 +0000 UTC m=+182.490575959" watchObservedRunningTime="2026-09-09 20:55:21.449247794 +0000 UTC m=+182.491247604" Sep 09 20:55:24.779806 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:24.779757 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:55:32.352445 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:32.352411 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:32.453588 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:32.453550 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:32.553387 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:32.553347 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:47.691242 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:47.691203 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:51.273645 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:51.273608 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:58.856481 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:58.856442 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:55:58.932786 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:55:58.932735 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:56:19.780163 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:56:19.780062 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:19.796097 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:56:19.796067 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:20.679448 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:56:20.679422 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 20:56:28.468181 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:56:28.468139 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:56:28.572902 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:56:28.572868 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:56:33.969944 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:56:33.969893 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:56:34.035877 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:56:34.035837 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:56:40.378775 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:56:40.378740 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:56:40.486659 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:56:40.486620 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:56:44.688659 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:56:44.688610 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:56:44.870783 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:56:44.870747 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:56:46.179493 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:56:46.179455 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:56:46.273236 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:56:46.273203 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:57:19.362080 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:57:19.362052 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-qlzzw_247631db-a55e-4fbc-9bde-f921c2bb309c/console-operator/1.log" Sep 09 20:57:19.367268 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:57:19.367245 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-qlzzw_247631db-a55e-4fbc-9bde-f921c2bb309c/console-operator/1.log" Sep 09 20:57:19.368920 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:57:19.368897 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crjwd_d55a4588-7960-4998-a99e-fca8f803b053/ovn-acl-logging/0.log" Sep 09 20:57:19.372708 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:57:19.372686 2562 kubelet.go:1648] "Image garbage collection succeeded" Sep 09 20:57:19.375116 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:57:19.375093 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crjwd_d55a4588-7960-4998-a99e-fca8f803b053/ovn-acl-logging/0.log" Sep 09 20:57:47.417170 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:57:47.417136 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:57:48.917891 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:57:48.917848 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:13.251469 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:58:13.251379 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:13.469125 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:58:13.469082 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:21.131360 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:58:21.131321 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:21.612949 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:58:21.612898 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:21.902127 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:58:21.902037 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:22.199052 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:58:22.198952 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:22.451647 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:58:22.451554 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:22.910722 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:58:22.910690 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:23.246480 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:58:23.246392 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:23.842099 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:58:23.842057 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:24.073367 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:58:24.073332 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:24.248799 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:58:24.248687 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:24.522949 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:58:24.522842 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:48.465370 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:58:48.465331 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:59.159309 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:58:59.159271 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:59.257029 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:58:59.256991 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:58:59.357249 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:58:59.357216 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:13.436222 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:59:13.436187 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:13.533747 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:59:13.533711 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:45.897221 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:59:45.897169 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:45.987539 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:59:45.987494 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:46.081029 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:59:46.080976 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:46.182180 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:59:46.182091 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:46.281201 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:59:46.281164 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:46.520316 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:59:46.520205 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:46.609873 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:59:46.609824 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 20:59:46.669735 ip-10-0-136-126 kubenswrapper[2562]: I0909 20:59:46.669699 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:14.560454 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:00:14.560411 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:14.652530 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:00:14.652486 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:30.535450 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:00:30.535410 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:30.612381 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:00:30.612349 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:00:37.269685 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:00:37.269645 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:04.685403 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:04.685362 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:04.798138 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:04.798108 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:04.899036 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:04.898992 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:05.010278 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:05.010178 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:05.126534 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:05.126490 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:05.952943 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:05.952905 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:06.054026 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:06.053976 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:01:21.338447 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:21.338366 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-qxvhf"] Sep 09 21:01:21.341704 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:21.341682 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qxvhf" Sep 09 21:01:21.344435 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:21.344411 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\"" Sep 09 21:01:21.344435 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:21.344415 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Sep 09 21:01:21.344656 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:21.344479 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Sep 09 21:01:21.344656 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:21.344494 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Sep 09 21:01:21.345490 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:21.345473 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-jq25d\"" Sep 09 21:01:21.352585 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:21.352564 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-qxvhf"] Sep 09 21:01:21.444441 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:21.444405 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld2jq\" (UniqueName: \"kubernetes.io/projected/53b7c008-6ea4-43f1-a083-29e6480cebd4-kube-api-access-ld2jq\") pod \"s3-init-qxvhf\" (UID: \"53b7c008-6ea4-43f1-a083-29e6480cebd4\") " pod="kserve/s3-init-qxvhf" Sep 09 21:01:21.444629 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:21.444481 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53b7c008-6ea4-43f1-a083-29e6480cebd4-model-cache\") pod \"s3-init-qxvhf\" (UID: \"53b7c008-6ea4-43f1-a083-29e6480cebd4\") " pod="kserve/s3-init-qxvhf" Sep 09 21:01:21.545759 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:21.545726 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53b7c008-6ea4-43f1-a083-29e6480cebd4-model-cache\") pod \"s3-init-qxvhf\" (UID: \"53b7c008-6ea4-43f1-a083-29e6480cebd4\") " pod="kserve/s3-init-qxvhf" Sep 09 21:01:21.545960 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:21.545777 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld2jq\" (UniqueName: \"kubernetes.io/projected/53b7c008-6ea4-43f1-a083-29e6480cebd4-kube-api-access-ld2jq\") pod \"s3-init-qxvhf\" (UID: \"53b7c008-6ea4-43f1-a083-29e6480cebd4\") " pod="kserve/s3-init-qxvhf" Sep 09 21:01:21.546230 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:21.546208 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53b7c008-6ea4-43f1-a083-29e6480cebd4-model-cache\") pod \"s3-init-qxvhf\" (UID: \"53b7c008-6ea4-43f1-a083-29e6480cebd4\") " pod="kserve/s3-init-qxvhf" Sep 09 21:01:21.554991 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:21.554959 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld2jq\" (UniqueName: \"kubernetes.io/projected/53b7c008-6ea4-43f1-a083-29e6480cebd4-kube-api-access-ld2jq\") pod \"s3-init-qxvhf\" (UID: \"53b7c008-6ea4-43f1-a083-29e6480cebd4\") " pod="kserve/s3-init-qxvhf" Sep 09 21:01:21.661728 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:21.661627 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qxvhf" Sep 09 21:01:21.948700 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:21.948675 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-qxvhf"] Sep 09 21:01:21.951747 ip-10-0-136-126 kubenswrapper[2562]: W0909 21:01:21.951718 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53b7c008_6ea4_43f1_a083_29e6480cebd4.slice/crio-36ffa0c9ffb4110f88a4b57aa150fc50c75a14dc54bea31b8b3bc0fda2963673 WatchSource:0}: Error finding container 36ffa0c9ffb4110f88a4b57aa150fc50c75a14dc54bea31b8b3bc0fda2963673: Status 404 returned error can't find the container with id 36ffa0c9ffb4110f88a4b57aa150fc50c75a14dc54bea31b8b3bc0fda2963673 Sep 09 21:01:21.954022 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:21.954001 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 21:01:22.841825 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:22.841779 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qxvhf" event={"ID":"53b7c008-6ea4-43f1-a083-29e6480cebd4","Type":"ContainerStarted","Data":"36ffa0c9ffb4110f88a4b57aa150fc50c75a14dc54bea31b8b3bc0fda2963673"} Sep 09 21:01:26.862571 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:26.862533 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qxvhf" event={"ID":"53b7c008-6ea4-43f1-a083-29e6480cebd4","Type":"ContainerStarted","Data":"b034488630f6e084824e60654acf29a4d62d35f099fe4cc4424024779869f6f3"} Sep 09 21:01:27.873911 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:27.873872 2562 generic.go:356] "Generic (PLEG): container finished" podID="53b7c008-6ea4-43f1-a083-29e6480cebd4" containerID="b034488630f6e084824e60654acf29a4d62d35f099fe4cc4424024779869f6f3" exitCode=0 Sep 09 21:01:27.874341 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:27.873960 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qxvhf" event={"ID":"53b7c008-6ea4-43f1-a083-29e6480cebd4","Type":"ContainerDied","Data":"b034488630f6e084824e60654acf29a4d62d35f099fe4cc4424024779869f6f3"} Sep 09 21:01:32.900553 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:32.900511 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qxvhf" event={"ID":"53b7c008-6ea4-43f1-a083-29e6480cebd4","Type":"ContainerStarted","Data":"0745885f9905a94fc9b4f6aa5bdcc6dd14ecf051b675e7d26ee962393d6b4599"} Sep 09 21:01:32.926525 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:01:32.926460 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-qxvhf" podStartSLOduration=1.683129332 podStartE2EDuration="11.926437395s" totalImagesPullingTime="8.461756688s" totalInitContainerRuntime="1.781551375s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:32.925156996 +0000 UTC m=+553.967156804" watchObservedRunningTime="2026-09-09 21:01:32.926437395 +0000 UTC m=+553.968437205" Sep 09 21:02:19.390233 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:19.390206 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-qlzzw_247631db-a55e-4fbc-9bde-f921c2bb309c/console-operator/1.log" Sep 09 21:02:19.394264 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:19.394234 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-qlzzw_247631db-a55e-4fbc-9bde-f921c2bb309c/console-operator/1.log" Sep 09 21:02:19.396811 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:19.396791 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crjwd_d55a4588-7960-4998-a99e-fca8f803b053/ovn-acl-logging/0.log" Sep 09 21:02:19.400919 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:19.400897 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crjwd_d55a4588-7960-4998-a99e-fca8f803b053/ovn-acl-logging/0.log" Sep 09 21:02:38.160571 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:38.160537 2562 generic.go:356] "Generic (PLEG): container finished" podID="53b7c008-6ea4-43f1-a083-29e6480cebd4" containerID="0745885f9905a94fc9b4f6aa5bdcc6dd14ecf051b675e7d26ee962393d6b4599" exitCode=0 Sep 09 21:02:38.160975 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:38.160609 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qxvhf" event={"ID":"53b7c008-6ea4-43f1-a083-29e6480cebd4","Type":"ContainerDied","Data":"0745885f9905a94fc9b4f6aa5bdcc6dd14ecf051b675e7d26ee962393d6b4599"} Sep 09 21:02:39.290649 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:39.290618 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qxvhf" Sep 09 21:02:39.428758 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:39.428665 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ld2jq\" (UniqueName: \"kubernetes.io/projected/53b7c008-6ea4-43f1-a083-29e6480cebd4-kube-api-access-ld2jq\") pod \"53b7c008-6ea4-43f1-a083-29e6480cebd4\" (UID: \"53b7c008-6ea4-43f1-a083-29e6480cebd4\") " Sep 09 21:02:39.428758 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:39.428712 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53b7c008-6ea4-43f1-a083-29e6480cebd4-model-cache\") pod \"53b7c008-6ea4-43f1-a083-29e6480cebd4\" (UID: \"53b7c008-6ea4-43f1-a083-29e6480cebd4\") " Sep 09 21:02:39.430948 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:39.430897 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b7c008-6ea4-43f1-a083-29e6480cebd4-kube-api-access-ld2jq" (OuterVolumeSpecName: "kube-api-access-ld2jq") pod "53b7c008-6ea4-43f1-a083-29e6480cebd4" (UID: "53b7c008-6ea4-43f1-a083-29e6480cebd4"). InnerVolumeSpecName "kube-api-access-ld2jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 21:02:39.447714 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:39.447674 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b7c008-6ea4-43f1-a083-29e6480cebd4-model-cache" (OuterVolumeSpecName: "model-cache") pod "53b7c008-6ea4-43f1-a083-29e6480cebd4" (UID: "53b7c008-6ea4-43f1-a083-29e6480cebd4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 21:02:39.529926 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:39.529882 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53b7c008-6ea4-43f1-a083-29e6480cebd4-model-cache\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 21:02:39.529926 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:39.529917 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld2jq\" (UniqueName: \"kubernetes.io/projected/53b7c008-6ea4-43f1-a083-29e6480cebd4-kube-api-access-ld2jq\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 21:02:40.170005 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:40.169963 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qxvhf" event={"ID":"53b7c008-6ea4-43f1-a083-29e6480cebd4","Type":"ContainerDied","Data":"36ffa0c9ffb4110f88a4b57aa150fc50c75a14dc54bea31b8b3bc0fda2963673"} Sep 09 21:02:40.170005 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:40.169991 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qxvhf" Sep 09 21:02:40.170005 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:40.170009 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36ffa0c9ffb4110f88a4b57aa150fc50c75a14dc54bea31b8b3bc0fda2963673" Sep 09 21:02:40.595166 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:40.595127 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:40.683017 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:40.682988 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:52.653997 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:52.653925 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:52.868082 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:52.868042 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv"] Sep 09 21:02:52.868523 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:52.868504 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53b7c008-6ea4-43f1-a083-29e6480cebd4" containerName="s3-init" Sep 09 21:02:52.868581 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:52.868528 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="53b7c008-6ea4-43f1-a083-29e6480cebd4" containerName="s3-init" Sep 09 21:02:52.868581 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:52.868553 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53b7c008-6ea4-43f1-a083-29e6480cebd4" containerName="download-hf-model" Sep 09 21:02:52.868581 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:52.868562 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="53b7c008-6ea4-43f1-a083-29e6480cebd4" containerName="download-hf-model" Sep 09 21:02:52.868682 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:52.868665 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="53b7c008-6ea4-43f1-a083-29e6480cebd4" containerName="s3-init" Sep 09 21:02:52.872133 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:52.872106 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" Sep 09 21:02:52.877139 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:52.876390 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 09 21:02:52.877139 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:52.876724 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 09 21:02:52.877139 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:52.876969 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\"" Sep 09 21:02:52.877527 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:52.877506 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tlmnz\"" Sep 09 21:02:52.878008 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:52.877986 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 09 21:02:52.878217 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:52.878194 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\"" Sep 09 21:02:52.878501 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:52.878482 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 09 21:02:52.891702 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:52.887983 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv"] Sep 09 21:02:52.941488 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:52.941380 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/730d1e4a-566a-4916-b3c1-0db96834a7c8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-5xfrv\" (UID: \"730d1e4a-566a-4916-b3c1-0db96834a7c8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" Sep 09 21:02:52.941488 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:52.941431 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/730d1e4a-566a-4916-b3c1-0db96834a7c8-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-5xfrv\" (UID: \"730d1e4a-566a-4916-b3c1-0db96834a7c8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" Sep 09 21:02:52.941722 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:52.941522 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzv6s\" (UniqueName: \"kubernetes.io/projected/730d1e4a-566a-4916-b3c1-0db96834a7c8-kube-api-access-bzv6s\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-5xfrv\" (UID: \"730d1e4a-566a-4916-b3c1-0db96834a7c8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" Sep 09 21:02:52.941722 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:52.941589 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/730d1e4a-566a-4916-b3c1-0db96834a7c8-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-5xfrv\" (UID: \"730d1e4a-566a-4916-b3c1-0db96834a7c8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" Sep 09 21:02:52.941722 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:52.941638 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/730d1e4a-566a-4916-b3c1-0db96834a7c8-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-5xfrv\" (UID: \"730d1e4a-566a-4916-b3c1-0db96834a7c8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" Sep 09 21:02:53.042782 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:53.042728 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/730d1e4a-566a-4916-b3c1-0db96834a7c8-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-5xfrv\" (UID: \"730d1e4a-566a-4916-b3c1-0db96834a7c8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" Sep 09 21:02:53.043011 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:53.042808 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzv6s\" (UniqueName: \"kubernetes.io/projected/730d1e4a-566a-4916-b3c1-0db96834a7c8-kube-api-access-bzv6s\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-5xfrv\" (UID: \"730d1e4a-566a-4916-b3c1-0db96834a7c8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" Sep 09 21:02:53.043011 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:53.042864 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/730d1e4a-566a-4916-b3c1-0db96834a7c8-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-5xfrv\" (UID: \"730d1e4a-566a-4916-b3c1-0db96834a7c8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" Sep 09 21:02:53.043011 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:53.042911 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/730d1e4a-566a-4916-b3c1-0db96834a7c8-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-5xfrv\" (UID: \"730d1e4a-566a-4916-b3c1-0db96834a7c8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" Sep 09 21:02:53.043184 ip-10-0-136-126 kubenswrapper[2562]: E0909 21:02:53.043039 2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found Sep 09 21:02:53.043321 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:53.043296 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/730d1e4a-566a-4916-b3c1-0db96834a7c8-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-5xfrv\" (UID: \"730d1e4a-566a-4916-b3c1-0db96834a7c8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" Sep 09 21:02:53.043380 ip-10-0-136-126 kubenswrapper[2562]: E0909 21:02:53.043373 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/730d1e4a-566a-4916-b3c1-0db96834a7c8-proxy-tls podName:730d1e4a-566a-4916-b3c1-0db96834a7c8 nodeName:}" failed. No retries permitted until 2026-09-09 21:02:53.543351583 +0000 UTC m=+634.585351376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/730d1e4a-566a-4916-b3c1-0db96834a7c8-proxy-tls") pod "isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" (UID: "730d1e4a-566a-4916-b3c1-0db96834a7c8") : secret "isvc-canary-rollback-predictor-serving-cert" not found Sep 09 21:02:53.043442 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:53.043428 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/730d1e4a-566a-4916-b3c1-0db96834a7c8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-5xfrv\" (UID: \"730d1e4a-566a-4916-b3c1-0db96834a7c8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" Sep 09 21:02:53.043590 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:53.043566 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/730d1e4a-566a-4916-b3c1-0db96834a7c8-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-5xfrv\" (UID: \"730d1e4a-566a-4916-b3c1-0db96834a7c8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" Sep 09 21:02:53.044067 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:53.044048 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/730d1e4a-566a-4916-b3c1-0db96834a7c8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-5xfrv\" (UID: \"730d1e4a-566a-4916-b3c1-0db96834a7c8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" Sep 09 21:02:53.052418 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:53.052388 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzv6s\" (UniqueName: \"kubernetes.io/projected/730d1e4a-566a-4916-b3c1-0db96834a7c8-kube-api-access-bzv6s\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-5xfrv\" (UID: \"730d1e4a-566a-4916-b3c1-0db96834a7c8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" Sep 09 21:02:53.547235 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:53.547195 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/730d1e4a-566a-4916-b3c1-0db96834a7c8-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-5xfrv\" (UID: \"730d1e4a-566a-4916-b3c1-0db96834a7c8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" Sep 09 21:02:53.549546 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:53.549521 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/730d1e4a-566a-4916-b3c1-0db96834a7c8-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-5xfrv\" (UID: \"730d1e4a-566a-4916-b3c1-0db96834a7c8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" Sep 09 21:02:53.790009 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:53.789967 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" Sep 09 21:02:54.060708 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:54.060631 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv"] Sep 09 21:02:54.063438 ip-10-0-136-126 kubenswrapper[2562]: W0909 21:02:54.063411 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730d1e4a_566a_4916_b3c1_0db96834a7c8.slice/crio-3cd64b8216fe3cb94f258175ce96a2c6cf6f5de7c6d79cd9c844b650b7739175 WatchSource:0}: Error finding container 3cd64b8216fe3cb94f258175ce96a2c6cf6f5de7c6d79cd9c844b650b7739175: Status 404 returned error can't find the container with id 3cd64b8216fe3cb94f258175ce96a2c6cf6f5de7c6d79cd9c844b650b7739175 Sep 09 21:02:54.228219 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:54.228176 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" event={"ID":"730d1e4a-566a-4916-b3c1-0db96834a7c8","Type":"ContainerStarted","Data":"551d9967c16c01affde15582eda5b1e041d44a6c90c5ea4fe397c58852fb556d"} Sep 09 21:02:54.228396 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:54.228254 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" event={"ID":"730d1e4a-566a-4916-b3c1-0db96834a7c8","Type":"ContainerStarted","Data":"3cd64b8216fe3cb94f258175ce96a2c6cf6f5de7c6d79cd9c844b650b7739175"} Sep 09 21:02:54.533917 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:54.533875 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:02:58.245445 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:58.245409 2562 generic.go:356] "Generic (PLEG): container finished" podID="730d1e4a-566a-4916-b3c1-0db96834a7c8" containerID="551d9967c16c01affde15582eda5b1e041d44a6c90c5ea4fe397c58852fb556d" exitCode=0 Sep 09 21:02:58.245827 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:02:58.245483 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" event={"ID":"730d1e4a-566a-4916-b3c1-0db96834a7c8","Type":"ContainerDied","Data":"551d9967c16c01affde15582eda5b1e041d44a6c90c5ea4fe397c58852fb556d"} Sep 09 21:03:12.320366 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:03:12.320329 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" event={"ID":"730d1e4a-566a-4916-b3c1-0db96834a7c8","Type":"ContainerStarted","Data":"dda4819a487f6b5622e0d1265148efbfc8ba8e8e1a7f69c45cf2f853edc42396"} Sep 09 21:03:14.331655 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:03:14.331612 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" event={"ID":"730d1e4a-566a-4916-b3c1-0db96834a7c8","Type":"ContainerStarted","Data":"2e086ca9b39161b4ebb1ae948315151ee5f9d87ff4d51bddf4196116dd04cd11"} Sep 09 21:03:14.332030 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:03:14.331733 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" Sep 09 21:03:14.354042 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:03:14.353984 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" podStartSLOduration=2.969348823 podStartE2EDuration="22.353955136s" totalImagesPullingTime="15.506211961s" totalInitContainerRuntime="3.878394352s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:03:14.351949929 +0000 UTC m=+655.393949731" watchObservedRunningTime="2026-09-09 21:03:14.353955136 +0000 UTC m=+655.395954939" Sep 09 21:03:15.335996 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:03:15.335965 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" Sep 09 21:03:15.337290 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:03:15.337260 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" podUID="730d1e4a-566a-4916-b3c1-0db96834a7c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Sep 09 21:03:16.340337 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:03:16.340299 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" podUID="730d1e4a-566a-4916-b3c1-0db96834a7c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Sep 09 21:03:21.345764 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:03:21.345736 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" Sep 09 21:03:21.346260 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:03:21.346235 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" podUID="730d1e4a-566a-4916-b3c1-0db96834a7c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Sep 09 21:03:31.346490 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:03:31.346438 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" podUID="730d1e4a-566a-4916-b3c1-0db96834a7c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Sep 09 21:03:41.347202 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:03:41.347155 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" podUID="730d1e4a-566a-4916-b3c1-0db96834a7c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Sep 09 21:03:51.346507 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:03:51.346463 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" podUID="730d1e4a-566a-4916-b3c1-0db96834a7c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Sep 09 21:04:01.346906 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:04:01.346864 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" podUID="730d1e4a-566a-4916-b3c1-0db96834a7c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Sep 09 21:04:11.346385 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:04:11.346293 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" podUID="730d1e4a-566a-4916-b3c1-0db96834a7c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Sep 09 21:04:16.349453 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:04:16.349415 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:04:16.415131 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:04:16.415094 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:04:21.346658 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:04:21.346615 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" podUID="730d1e4a-566a-4916-b3c1-0db96834a7c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Sep 09 21:04:31.347791 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:04:31.347758 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" Sep 09 21:04:42.616394 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:04:42.616354 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:04:47.811154 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:04:47.811117 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:04:47.927609 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:04:47.927573 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:04:48.186221 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:04:48.186181 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:04:48.311085 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:04:48.311045 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:04:48.407143 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:04:48.407091 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:05:07.073051 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:05:07.073018 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:05:21.996085 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:05:21.996051 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:06:12.766280 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:12.766243 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv"] Sep 09 21:06:12.766855 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:12.766569 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" podUID="730d1e4a-566a-4916-b3c1-0db96834a7c8" containerName="kserve-container" containerID="cri-o://dda4819a487f6b5622e0d1265148efbfc8ba8e8e1a7f69c45cf2f853edc42396" gracePeriod=30 Sep 09 21:06:12.766855 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:12.766656 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" podUID="730d1e4a-566a-4916-b3c1-0db96834a7c8" containerName="kube-rbac-proxy" containerID="cri-o://2e086ca9b39161b4ebb1ae948315151ee5f9d87ff4d51bddf4196116dd04cd11" gracePeriod=30 Sep 09 21:06:12.791052 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:12.791009 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:06:13.057300 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:13.057216 2562 generic.go:356] "Generic (PLEG): container finished" podID="730d1e4a-566a-4916-b3c1-0db96834a7c8" containerID="2e086ca9b39161b4ebb1ae948315151ee5f9d87ff4d51bddf4196116dd04cd11" exitCode=2 Sep 09 21:06:13.057300 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:13.057282 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" event={"ID":"730d1e4a-566a-4916-b3c1-0db96834a7c8","Type":"ContainerDied","Data":"2e086ca9b39161b4ebb1ae948315151ee5f9d87ff4d51bddf4196116dd04cd11"} Sep 09 21:06:16.341172 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:16.341124 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" podUID="730d1e4a-566a-4916-b3c1-0db96834a7c8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused" Sep 09 21:06:17.815129 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:17.815104 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" Sep 09 21:06:17.835727 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:17.835690 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/730d1e4a-566a-4916-b3c1-0db96834a7c8-kserve-provision-location\") pod \"730d1e4a-566a-4916-b3c1-0db96834a7c8\" (UID: \"730d1e4a-566a-4916-b3c1-0db96834a7c8\") " Sep 09 21:06:17.835905 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:17.835737 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/730d1e4a-566a-4916-b3c1-0db96834a7c8-kube-rbac-proxy-sar-config\") pod \"730d1e4a-566a-4916-b3c1-0db96834a7c8\" (UID: \"730d1e4a-566a-4916-b3c1-0db96834a7c8\") " Sep 09 21:06:17.835905 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:17.835754 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzv6s\" (UniqueName: \"kubernetes.io/projected/730d1e4a-566a-4916-b3c1-0db96834a7c8-kube-api-access-bzv6s\") pod \"730d1e4a-566a-4916-b3c1-0db96834a7c8\" (UID: \"730d1e4a-566a-4916-b3c1-0db96834a7c8\") " Sep 09 21:06:17.835905 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:17.835786 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/730d1e4a-566a-4916-b3c1-0db96834a7c8-cabundle-cert\") pod \"730d1e4a-566a-4916-b3c1-0db96834a7c8\" (UID: \"730d1e4a-566a-4916-b3c1-0db96834a7c8\") " Sep 09 21:06:17.835905 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:17.835847 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/730d1e4a-566a-4916-b3c1-0db96834a7c8-proxy-tls\") pod \"730d1e4a-566a-4916-b3c1-0db96834a7c8\" (UID: \"730d1e4a-566a-4916-b3c1-0db96834a7c8\") " Sep 09 21:06:17.836148 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:17.836086 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730d1e4a-566a-4916-b3c1-0db96834a7c8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "730d1e4a-566a-4916-b3c1-0db96834a7c8" (UID: "730d1e4a-566a-4916-b3c1-0db96834a7c8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 21:06:17.836148 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:17.836127 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730d1e4a-566a-4916-b3c1-0db96834a7c8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "730d1e4a-566a-4916-b3c1-0db96834a7c8" (UID: "730d1e4a-566a-4916-b3c1-0db96834a7c8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 21:06:17.836258 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:17.836234 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730d1e4a-566a-4916-b3c1-0db96834a7c8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "730d1e4a-566a-4916-b3c1-0db96834a7c8" (UID: "730d1e4a-566a-4916-b3c1-0db96834a7c8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 21:06:17.838111 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:17.838084 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/730d1e4a-566a-4916-b3c1-0db96834a7c8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "730d1e4a-566a-4916-b3c1-0db96834a7c8" (UID: "730d1e4a-566a-4916-b3c1-0db96834a7c8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 21:06:17.838224 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:17.838143 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730d1e4a-566a-4916-b3c1-0db96834a7c8-kube-api-access-bzv6s" (OuterVolumeSpecName: "kube-api-access-bzv6s") pod "730d1e4a-566a-4916-b3c1-0db96834a7c8" (UID: "730d1e4a-566a-4916-b3c1-0db96834a7c8"). InnerVolumeSpecName "kube-api-access-bzv6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 21:06:17.936874 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:17.936786 2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/730d1e4a-566a-4916-b3c1-0db96834a7c8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 21:06:17.936874 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:17.936813 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzv6s\" (UniqueName: \"kubernetes.io/projected/730d1e4a-566a-4916-b3c1-0db96834a7c8-kube-api-access-bzv6s\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 21:06:17.936874 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:17.936825 2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/730d1e4a-566a-4916-b3c1-0db96834a7c8-cabundle-cert\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 21:06:17.936874 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:17.936834 2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/730d1e4a-566a-4916-b3c1-0db96834a7c8-proxy-tls\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 21:06:17.936874 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:17.936842 2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/730d1e4a-566a-4916-b3c1-0db96834a7c8-kserve-provision-location\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 21:06:17.944420 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:17.944389 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:06:18.033547 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.033507 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:06:18.065802 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.065769 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk"] Sep 09 21:06:18.066121 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.066107 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="730d1e4a-566a-4916-b3c1-0db96834a7c8" containerName="kserve-container" Sep 09 21:06:18.066174 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.066122 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="730d1e4a-566a-4916-b3c1-0db96834a7c8" containerName="kserve-container" Sep 09 21:06:18.066174 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.066136 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="730d1e4a-566a-4916-b3c1-0db96834a7c8" containerName="kube-rbac-proxy" Sep 09 21:06:18.066174 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.066142 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="730d1e4a-566a-4916-b3c1-0db96834a7c8" containerName="kube-rbac-proxy" Sep 09 21:06:18.066174 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.066150 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="730d1e4a-566a-4916-b3c1-0db96834a7c8" containerName="storage-initializer" Sep 09 21:06:18.066174 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.066156 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="730d1e4a-566a-4916-b3c1-0db96834a7c8" containerName="storage-initializer" Sep 09 21:06:18.066323 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.066204 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="730d1e4a-566a-4916-b3c1-0db96834a7c8" containerName="kube-rbac-proxy" Sep 09 21:06:18.066323 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.066215 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="730d1e4a-566a-4916-b3c1-0db96834a7c8" containerName="kserve-container" Sep 09 21:06:18.069339 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.069319 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" Sep 09 21:06:18.071881 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.071863 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\"" Sep 09 21:06:18.072013 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.071900 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\"" Sep 09 21:06:18.084958 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.084378 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk"] Sep 09 21:06:18.088683 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.088651 2562 generic.go:356] "Generic (PLEG): container finished" podID="730d1e4a-566a-4916-b3c1-0db96834a7c8" containerID="dda4819a487f6b5622e0d1265148efbfc8ba8e8e1a7f69c45cf2f853edc42396" exitCode=0 Sep 09 21:06:18.088896 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.088687 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" event={"ID":"730d1e4a-566a-4916-b3c1-0db96834a7c8","Type":"ContainerDied","Data":"dda4819a487f6b5622e0d1265148efbfc8ba8e8e1a7f69c45cf2f853edc42396"} Sep 09 21:06:18.089045 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.088953 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" event={"ID":"730d1e4a-566a-4916-b3c1-0db96834a7c8","Type":"ContainerDied","Data":"3cd64b8216fe3cb94f258175ce96a2c6cf6f5de7c6d79cd9c844b650b7739175"} Sep 09 21:06:18.089045 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.089001 2562 scope.go:117] "RemoveContainer" containerID="2e086ca9b39161b4ebb1ae948315151ee5f9d87ff4d51bddf4196116dd04cd11" Sep 09 21:06:18.089169 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.088785 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv" Sep 09 21:06:18.099707 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.099683 2562 scope.go:117] "RemoveContainer" containerID="dda4819a487f6b5622e0d1265148efbfc8ba8e8e1a7f69c45cf2f853edc42396" Sep 09 21:06:18.108056 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.108039 2562 scope.go:117] "RemoveContainer" containerID="551d9967c16c01affde15582eda5b1e041d44a6c90c5ea4fe397c58852fb556d" Sep 09 21:06:18.115700 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.115678 2562 scope.go:117] "RemoveContainer" containerID="2e086ca9b39161b4ebb1ae948315151ee5f9d87ff4d51bddf4196116dd04cd11" Sep 09 21:06:18.116023 ip-10-0-136-126 kubenswrapper[2562]: E0909 21:06:18.116002 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e086ca9b39161b4ebb1ae948315151ee5f9d87ff4d51bddf4196116dd04cd11\": container with ID starting with 2e086ca9b39161b4ebb1ae948315151ee5f9d87ff4d51bddf4196116dd04cd11 not found: ID does not exist" containerID="2e086ca9b39161b4ebb1ae948315151ee5f9d87ff4d51bddf4196116dd04cd11" Sep 09 21:06:18.116088 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.116033 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e086ca9b39161b4ebb1ae948315151ee5f9d87ff4d51bddf4196116dd04cd11"} err="failed to get container status \"2e086ca9b39161b4ebb1ae948315151ee5f9d87ff4d51bddf4196116dd04cd11\": rpc error: code = NotFound desc = could not find container \"2e086ca9b39161b4ebb1ae948315151ee5f9d87ff4d51bddf4196116dd04cd11\": container with ID starting with 2e086ca9b39161b4ebb1ae948315151ee5f9d87ff4d51bddf4196116dd04cd11 not found: ID does not exist" Sep 09 21:06:18.116088 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.116054 2562 scope.go:117] "RemoveContainer" containerID="dda4819a487f6b5622e0d1265148efbfc8ba8e8e1a7f69c45cf2f853edc42396" Sep 09 21:06:18.116288 ip-10-0-136-126 kubenswrapper[2562]: E0909 21:06:18.116272 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dda4819a487f6b5622e0d1265148efbfc8ba8e8e1a7f69c45cf2f853edc42396\": container with ID starting with dda4819a487f6b5622e0d1265148efbfc8ba8e8e1a7f69c45cf2f853edc42396 not found: ID does not exist" containerID="dda4819a487f6b5622e0d1265148efbfc8ba8e8e1a7f69c45cf2f853edc42396" Sep 09 21:06:18.116347 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.116292 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dda4819a487f6b5622e0d1265148efbfc8ba8e8e1a7f69c45cf2f853edc42396"} err="failed to get container status \"dda4819a487f6b5622e0d1265148efbfc8ba8e8e1a7f69c45cf2f853edc42396\": rpc error: code = NotFound desc = could not find container \"dda4819a487f6b5622e0d1265148efbfc8ba8e8e1a7f69c45cf2f853edc42396\": container with ID starting with dda4819a487f6b5622e0d1265148efbfc8ba8e8e1a7f69c45cf2f853edc42396 not found: ID does not exist" Sep 09 21:06:18.116347 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.116306 2562 scope.go:117] "RemoveContainer" containerID="551d9967c16c01affde15582eda5b1e041d44a6c90c5ea4fe397c58852fb556d" Sep 09 21:06:18.116603 ip-10-0-136-126 kubenswrapper[2562]: E0909 21:06:18.116584 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"551d9967c16c01affde15582eda5b1e041d44a6c90c5ea4fe397c58852fb556d\": container with ID starting with 551d9967c16c01affde15582eda5b1e041d44a6c90c5ea4fe397c58852fb556d not found: ID does not exist" containerID="551d9967c16c01affde15582eda5b1e041d44a6c90c5ea4fe397c58852fb556d" Sep 09 21:06:18.116655 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.116611 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"551d9967c16c01affde15582eda5b1e041d44a6c90c5ea4fe397c58852fb556d"} err="failed to get container status \"551d9967c16c01affde15582eda5b1e041d44a6c90c5ea4fe397c58852fb556d\": rpc error: code = NotFound desc = could not find container \"551d9967c16c01affde15582eda5b1e041d44a6c90c5ea4fe397c58852fb556d\": container with ID starting with 551d9967c16c01affde15582eda5b1e041d44a6c90c5ea4fe397c58852fb556d not found: ID does not exist" Sep 09 21:06:18.119959 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.119920 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv"] Sep 09 21:06:18.124059 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.124031 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-5xfrv"] Sep 09 21:06:18.124989 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.124919 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:06:18.138218 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.138181 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8ffe01c-b06a-4073-9007-08d8fba212bd-proxy-tls\") pod \"isvc-canary-stop-predictor-59fbfd5bc-7dblk\" (UID: \"e8ffe01c-b06a-4073-9007-08d8fba212bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" Sep 09 21:06:18.138367 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.138228 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8ffe01c-b06a-4073-9007-08d8fba212bd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59fbfd5bc-7dblk\" (UID: \"e8ffe01c-b06a-4073-9007-08d8fba212bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" Sep 09 21:06:18.138367 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.138325 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wwk4\" (UniqueName: \"kubernetes.io/projected/e8ffe01c-b06a-4073-9007-08d8fba212bd-kube-api-access-7wwk4\") pod \"isvc-canary-stop-predictor-59fbfd5bc-7dblk\" (UID: \"e8ffe01c-b06a-4073-9007-08d8fba212bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" Sep 09 21:06:18.138367 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.138358 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8ffe01c-b06a-4073-9007-08d8fba212bd-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59fbfd5bc-7dblk\" (UID: \"e8ffe01c-b06a-4073-9007-08d8fba212bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" Sep 09 21:06:18.138524 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.138383 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8ffe01c-b06a-4073-9007-08d8fba212bd-cabundle-cert\") pod \"isvc-canary-stop-predictor-59fbfd5bc-7dblk\" (UID: \"e8ffe01c-b06a-4073-9007-08d8fba212bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" Sep 09 21:06:18.239596 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.239493 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8ffe01c-b06a-4073-9007-08d8fba212bd-proxy-tls\") pod \"isvc-canary-stop-predictor-59fbfd5bc-7dblk\" (UID: \"e8ffe01c-b06a-4073-9007-08d8fba212bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" Sep 09 21:06:18.239596 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.239541 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8ffe01c-b06a-4073-9007-08d8fba212bd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59fbfd5bc-7dblk\" (UID: \"e8ffe01c-b06a-4073-9007-08d8fba212bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" Sep 09 21:06:18.239817 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.239601 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wwk4\" (UniqueName: \"kubernetes.io/projected/e8ffe01c-b06a-4073-9007-08d8fba212bd-kube-api-access-7wwk4\") pod \"isvc-canary-stop-predictor-59fbfd5bc-7dblk\" (UID: \"e8ffe01c-b06a-4073-9007-08d8fba212bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" Sep 09 21:06:18.239817 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.239622 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8ffe01c-b06a-4073-9007-08d8fba212bd-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59fbfd5bc-7dblk\" (UID: \"e8ffe01c-b06a-4073-9007-08d8fba212bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" Sep 09 21:06:18.239817 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.239646 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8ffe01c-b06a-4073-9007-08d8fba212bd-cabundle-cert\") pod \"isvc-canary-stop-predictor-59fbfd5bc-7dblk\" (UID: \"e8ffe01c-b06a-4073-9007-08d8fba212bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" Sep 09 21:06:18.240203 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.240159 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8ffe01c-b06a-4073-9007-08d8fba212bd-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59fbfd5bc-7dblk\" (UID: \"e8ffe01c-b06a-4073-9007-08d8fba212bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" Sep 09 21:06:18.240366 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.240345 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8ffe01c-b06a-4073-9007-08d8fba212bd-cabundle-cert\") pod \"isvc-canary-stop-predictor-59fbfd5bc-7dblk\" (UID: \"e8ffe01c-b06a-4073-9007-08d8fba212bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" Sep 09 21:06:18.240460 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.240367 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8ffe01c-b06a-4073-9007-08d8fba212bd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59fbfd5bc-7dblk\" (UID: \"e8ffe01c-b06a-4073-9007-08d8fba212bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" Sep 09 21:06:18.242034 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.242014 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8ffe01c-b06a-4073-9007-08d8fba212bd-proxy-tls\") pod \"isvc-canary-stop-predictor-59fbfd5bc-7dblk\" (UID: \"e8ffe01c-b06a-4073-9007-08d8fba212bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" Sep 09 21:06:18.247475 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.247458 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wwk4\" (UniqueName: \"kubernetes.io/projected/e8ffe01c-b06a-4073-9007-08d8fba212bd-kube-api-access-7wwk4\") pod \"isvc-canary-stop-predictor-59fbfd5bc-7dblk\" (UID: \"e8ffe01c-b06a-4073-9007-08d8fba212bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" Sep 09 21:06:18.376174 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.376142 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:06:18.389122 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.389091 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" Sep 09 21:06:18.673018 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:18.672965 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk"] Sep 09 21:06:18.675207 ip-10-0-136-126 kubenswrapper[2562]: W0909 21:06:18.675178 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ffe01c_b06a_4073_9007_08d8fba212bd.slice/crio-90f5f5b087e676324743bb70e372b49321bf942f63c6ab80892670605eba0c92 WatchSource:0}: Error finding container 90f5f5b087e676324743bb70e372b49321bf942f63c6ab80892670605eba0c92: Status 404 returned error can't find the container with id 90f5f5b087e676324743bb70e372b49321bf942f63c6ab80892670605eba0c92 Sep 09 21:06:19.096436 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:19.096399 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" event={"ID":"e8ffe01c-b06a-4073-9007-08d8fba212bd","Type":"ContainerStarted","Data":"b82da19f22655fa0c79d4421ccbfd40f286914c3330a7de08acb145097055155"} Sep 09 21:06:19.096952 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:19.096478 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" event={"ID":"e8ffe01c-b06a-4073-9007-08d8fba212bd","Type":"ContainerStarted","Data":"90f5f5b087e676324743bb70e372b49321bf942f63c6ab80892670605eba0c92"} Sep 09 21:06:19.489358 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:19.489324 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="730d1e4a-566a-4916-b3c1-0db96834a7c8" path="/var/lib/kubelet/pods/730d1e4a-566a-4916-b3c1-0db96834a7c8/volumes" Sep 09 21:06:23.115054 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:23.115022 2562 generic.go:356] "Generic (PLEG): container finished" podID="e8ffe01c-b06a-4073-9007-08d8fba212bd" containerID="b82da19f22655fa0c79d4421ccbfd40f286914c3330a7de08acb145097055155" exitCode=0 Sep 09 21:06:23.115453 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:23.115081 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" event={"ID":"e8ffe01c-b06a-4073-9007-08d8fba212bd","Type":"ContainerDied","Data":"b82da19f22655fa0c79d4421ccbfd40f286914c3330a7de08acb145097055155"} Sep 09 21:06:24.123158 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:24.123123 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" event={"ID":"e8ffe01c-b06a-4073-9007-08d8fba212bd","Type":"ContainerStarted","Data":"7fe356e7ae497a244d1eb387bf7d2b45a9dc845be3b76829900256533d5f231a"} Sep 09 21:06:24.123592 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:24.123188 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" event={"ID":"e8ffe01c-b06a-4073-9007-08d8fba212bd","Type":"ContainerStarted","Data":"d987779843901d4b17492bf148c79f557a5eb52ccb324bfc40d86865e4a5a450"} Sep 09 21:06:24.123592 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:24.123399 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" Sep 09 21:06:24.140709 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:24.140657 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" podStartSLOduration=2.183392936 podStartE2EDuration="6.140641393s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.957248457s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:06:24.140009137 +0000 UTC m=+845.182008947" watchObservedRunningTime="2026-09-09 21:06:24.140641393 +0000 UTC m=+845.182641205" Sep 09 21:06:25.127731 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:25.127697 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" Sep 09 21:06:25.129093 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:25.129065 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" podUID="e8ffe01c-b06a-4073-9007-08d8fba212bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Sep 09 21:06:26.132134 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:26.132092 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" podUID="e8ffe01c-b06a-4073-9007-08d8fba212bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Sep 09 21:06:27.897541 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:27.897502 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:06:28.400607 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:28.400553 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:06:31.136789 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:31.136761 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" Sep 09 21:06:31.137339 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:31.137307 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" podUID="e8ffe01c-b06a-4073-9007-08d8fba212bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Sep 09 21:06:37.962715 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:37.962676 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:06:38.079399 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.079366 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:06:38.178863 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.178826 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:06:38.321551 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.321467 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:06:38.442066 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.442031 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl"] Sep 09 21:06:38.445555 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.445534 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" Sep 09 21:06:38.448058 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.448028 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-887b3-predictor-serving-cert\"" Sep 09 21:06:38.448489 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.448474 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-887b3-kube-rbac-proxy-sar-config\"" Sep 09 21:06:38.460080 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.460055 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:06:38.461187 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.461165 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl"] Sep 09 21:06:38.520519 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.520478 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl\" (UID: \"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" Sep 09 21:06:38.520704 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.520530 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl\" (UID: \"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" Sep 09 21:06:38.520704 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.520556 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwnp8\" (UniqueName: \"kubernetes.io/projected/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-kube-api-access-mwnp8\") pod \"isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl\" (UID: \"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" Sep 09 21:06:38.520704 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.520611 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl\" (UID: \"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" Sep 09 21:06:38.520704 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.520671 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl\" (UID: \"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" Sep 09 21:06:38.554408 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.554366 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:06:38.621470 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.621431 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl\" (UID: \"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" Sep 09 21:06:38.621697 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.621483 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl\" (UID: \"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" Sep 09 21:06:38.621697 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.621607 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwnp8\" (UniqueName: \"kubernetes.io/projected/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-kube-api-access-mwnp8\") pod \"isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl\" (UID: \"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" Sep 09 21:06:38.621697 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.621652 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl\" (UID: \"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" Sep 09 21:06:38.621880 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.621710 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl\" (UID: \"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" Sep 09 21:06:38.621980 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.621899 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl\" (UID: \"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" Sep 09 21:06:38.622182 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.622164 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl\" (UID: \"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" Sep 09 21:06:38.622224 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.622214 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl\" (UID: \"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" Sep 09 21:06:38.624208 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.624187 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl\" (UID: \"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" Sep 09 21:06:38.631433 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.631406 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwnp8\" (UniqueName: \"kubernetes.io/projected/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-kube-api-access-mwnp8\") pod \"isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl\" (UID: \"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" Sep 09 21:06:38.755620 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:38.755577 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" Sep 09 21:06:39.030477 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:39.030446 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl"] Sep 09 21:06:39.032880 ip-10-0-136-126 kubenswrapper[2562]: W0909 21:06:39.032848 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9870a4_53b7_4ad9_b3ed_49fb473fa8c0.slice/crio-92e6d32ce748632325dc57987884b3c312ae9c4afbbf34e6eab1a60c8fb6adb5 WatchSource:0}: Error finding container 92e6d32ce748632325dc57987884b3c312ae9c4afbbf34e6eab1a60c8fb6adb5: Status 404 returned error can't find the container with id 92e6d32ce748632325dc57987884b3c312ae9c4afbbf34e6eab1a60c8fb6adb5 Sep 09 21:06:39.188744 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:39.188708 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" event={"ID":"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0","Type":"ContainerStarted","Data":"a54ed759771062b0f1c792f9eb7d20e5a1a1db0ed53d3a965444d8051ae2c9c9"} Sep 09 21:06:39.188895 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:39.188770 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" event={"ID":"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0","Type":"ContainerStarted","Data":"92e6d32ce748632325dc57987884b3c312ae9c4afbbf34e6eab1a60c8fb6adb5"} Sep 09 21:06:41.138107 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:41.138061 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" podUID="e8ffe01c-b06a-4073-9007-08d8fba212bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Sep 09 21:06:43.209019 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:43.208983 2562 generic.go:356] "Generic (PLEG): container finished" podID="2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" containerID="a54ed759771062b0f1c792f9eb7d20e5a1a1db0ed53d3a965444d8051ae2c9c9" exitCode=0 Sep 09 21:06:43.209507 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:43.209061 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" event={"ID":"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0","Type":"ContainerDied","Data":"a54ed759771062b0f1c792f9eb7d20e5a1a1db0ed53d3a965444d8051ae2c9c9"} Sep 09 21:06:43.210575 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:43.210560 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 21:06:50.253660 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:50.253624 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:06:50.317554 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:50.317522 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:06:51.137443 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:06:51.137401 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" podUID="e8ffe01c-b06a-4073-9007-08d8fba212bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Sep 09 21:07:01.137705 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:01.137658 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" podUID="e8ffe01c-b06a-4073-9007-08d8fba212bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Sep 09 21:07:03.315129 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:03.315093 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" event={"ID":"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0","Type":"ContainerStarted","Data":"9ed727270fd0da190a4f98b4d1b5f84db79c351e467e13c3661d5c5911790843"} Sep 09 21:07:03.315515 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:03.315160 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" event={"ID":"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0","Type":"ContainerStarted","Data":"0881c823ff0dd46b669d4ab52c5dcfaea0ddffe314aebf368a54ff02b2a5ca1a"} Sep 09 21:07:03.315515 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:03.315360 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" Sep 09 21:07:03.337525 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:03.337468 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" podStartSLOduration=2.245446776 podStartE2EDuration="25.337449037s" totalImagesPullingTime="19.063965859s" totalInitContainerRuntime="4.028036402s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:07:03.336284367 +0000 UTC m=+884.378284199" watchObservedRunningTime="2026-09-09 21:07:03.337449037 +0000 UTC m=+884.379448846" Sep 09 21:07:04.320232 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:04.320195 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" Sep 09 21:07:04.321613 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:04.321585 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" podUID="2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Sep 09 21:07:05.325113 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:05.325074 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" podUID="2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Sep 09 21:07:10.330192 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:10.330161 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" Sep 09 21:07:10.330679 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:10.330585 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" podUID="2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Sep 09 21:07:11.138008 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:11.137888 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" podUID="e8ffe01c-b06a-4073-9007-08d8fba212bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Sep 09 21:07:19.421474 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:19.421446 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-qlzzw_247631db-a55e-4fbc-9bde-f921c2bb309c/console-operator/1.log" Sep 09 21:07:19.422032 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:19.422013 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-qlzzw_247631db-a55e-4fbc-9bde-f921c2bb309c/console-operator/1.log" Sep 09 21:07:19.428391 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:19.428366 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crjwd_d55a4588-7960-4998-a99e-fca8f803b053/ovn-acl-logging/0.log" Sep 09 21:07:19.429068 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:19.429048 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crjwd_d55a4588-7960-4998-a99e-fca8f803b053/ovn-acl-logging/0.log" Sep 09 21:07:20.331354 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:20.331312 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" podUID="2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Sep 09 21:07:21.138204 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:21.138162 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" podUID="e8ffe01c-b06a-4073-9007-08d8fba212bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Sep 09 21:07:30.331416 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:30.331374 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" podUID="2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Sep 09 21:07:31.137255 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:31.137216 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" podUID="e8ffe01c-b06a-4073-9007-08d8fba212bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Sep 09 21:07:40.331383 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:40.331341 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" podUID="2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Sep 09 21:07:41.137208 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:41.137162 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" podUID="e8ffe01c-b06a-4073-9007-08d8fba212bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Sep 09 21:07:41.196606 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:41.196558 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk"] Sep 09 21:07:41.196814 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:41.196793 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" podUID="e8ffe01c-b06a-4073-9007-08d8fba212bd" containerName="kserve-container" containerID="cri-o://d987779843901d4b17492bf148c79f557a5eb52ccb324bfc40d86865e4a5a450" gracePeriod=30 Sep 09 21:07:41.196956 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:41.196865 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" podUID="e8ffe01c-b06a-4073-9007-08d8fba212bd" containerName="kube-rbac-proxy" containerID="cri-o://7fe356e7ae497a244d1eb387bf7d2b45a9dc845be3b76829900256533d5f231a" gracePeriod=30 Sep 09 21:07:41.197133 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:41.197101 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" podUID="e8ffe01c-b06a-4073-9007-08d8fba212bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Sep 09 21:07:41.487631 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:41.487547 2562 generic.go:356] "Generic (PLEG): container finished" podID="e8ffe01c-b06a-4073-9007-08d8fba212bd" containerID="7fe356e7ae497a244d1eb387bf7d2b45a9dc845be3b76829900256533d5f231a" exitCode=2 Sep 09 21:07:41.489618 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:41.489595 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" event={"ID":"e8ffe01c-b06a-4073-9007-08d8fba212bd","Type":"ContainerDied","Data":"7fe356e7ae497a244d1eb387bf7d2b45a9dc845be3b76829900256533d5f231a"} Sep 09 21:07:46.133223 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:46.133176 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" podUID="e8ffe01c-b06a-4073-9007-08d8fba212bd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused" Sep 09 21:07:47.740319 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:47.740248 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" Sep 09 21:07:47.831387 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:47.831343 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8ffe01c-b06a-4073-9007-08d8fba212bd-cabundle-cert\") pod \"e8ffe01c-b06a-4073-9007-08d8fba212bd\" (UID: \"e8ffe01c-b06a-4073-9007-08d8fba212bd\") " Sep 09 21:07:47.831584 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:47.831400 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8ffe01c-b06a-4073-9007-08d8fba212bd-kserve-provision-location\") pod \"e8ffe01c-b06a-4073-9007-08d8fba212bd\" (UID: \"e8ffe01c-b06a-4073-9007-08d8fba212bd\") " Sep 09 21:07:47.831584 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:47.831436 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8ffe01c-b06a-4073-9007-08d8fba212bd-proxy-tls\") pod \"e8ffe01c-b06a-4073-9007-08d8fba212bd\" (UID: \"e8ffe01c-b06a-4073-9007-08d8fba212bd\") " Sep 09 21:07:47.831584 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:47.831455 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8ffe01c-b06a-4073-9007-08d8fba212bd-kube-rbac-proxy-sar-config\") pod \"e8ffe01c-b06a-4073-9007-08d8fba212bd\" (UID: \"e8ffe01c-b06a-4073-9007-08d8fba212bd\") " Sep 09 21:07:47.831584 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:47.831486 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wwk4\" (UniqueName: \"kubernetes.io/projected/e8ffe01c-b06a-4073-9007-08d8fba212bd-kube-api-access-7wwk4\") pod \"e8ffe01c-b06a-4073-9007-08d8fba212bd\" (UID: \"e8ffe01c-b06a-4073-9007-08d8fba212bd\") " Sep 09 21:07:47.831812 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:47.831783 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ffe01c-b06a-4073-9007-08d8fba212bd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e8ffe01c-b06a-4073-9007-08d8fba212bd" (UID: "e8ffe01c-b06a-4073-9007-08d8fba212bd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 21:07:47.831870 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:47.831820 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ffe01c-b06a-4073-9007-08d8fba212bd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e8ffe01c-b06a-4073-9007-08d8fba212bd" (UID: "e8ffe01c-b06a-4073-9007-08d8fba212bd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 21:07:47.831870 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:47.831820 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8ffe01c-b06a-4073-9007-08d8fba212bd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e8ffe01c-b06a-4073-9007-08d8fba212bd" (UID: "e8ffe01c-b06a-4073-9007-08d8fba212bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 21:07:47.833639 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:47.833619 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ffe01c-b06a-4073-9007-08d8fba212bd-kube-api-access-7wwk4" (OuterVolumeSpecName: "kube-api-access-7wwk4") pod "e8ffe01c-b06a-4073-9007-08d8fba212bd" (UID: "e8ffe01c-b06a-4073-9007-08d8fba212bd"). InnerVolumeSpecName "kube-api-access-7wwk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 21:07:47.833775 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:47.833756 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ffe01c-b06a-4073-9007-08d8fba212bd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e8ffe01c-b06a-4073-9007-08d8fba212bd" (UID: "e8ffe01c-b06a-4073-9007-08d8fba212bd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 21:07:47.932976 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:47.932911 2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8ffe01c-b06a-4073-9007-08d8fba212bd-cabundle-cert\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 21:07:47.932976 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:47.932971 2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8ffe01c-b06a-4073-9007-08d8fba212bd-kserve-provision-location\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 21:07:47.932976 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:47.932982 2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8ffe01c-b06a-4073-9007-08d8fba212bd-proxy-tls\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 21:07:47.933217 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:47.932991 2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8ffe01c-b06a-4073-9007-08d8fba212bd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 21:07:47.933217 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:47.932999 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wwk4\" (UniqueName: \"kubernetes.io/projected/e8ffe01c-b06a-4073-9007-08d8fba212bd-kube-api-access-7wwk4\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 21:07:48.001528 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:48.001445 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:07:48.102076 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:48.102043 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:07:48.529549 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:48.529514 2562 generic.go:356] "Generic (PLEG): container finished" podID="e8ffe01c-b06a-4073-9007-08d8fba212bd" containerID="d987779843901d4b17492bf148c79f557a5eb52ccb324bfc40d86865e4a5a450" exitCode=139 Sep 09 21:07:48.529754 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:48.529572 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" event={"ID":"e8ffe01c-b06a-4073-9007-08d8fba212bd","Type":"ContainerDied","Data":"d987779843901d4b17492bf148c79f557a5eb52ccb324bfc40d86865e4a5a450"} Sep 09 21:07:48.529754 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:48.529584 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" Sep 09 21:07:48.529754 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:48.529652 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk" event={"ID":"e8ffe01c-b06a-4073-9007-08d8fba212bd","Type":"ContainerDied","Data":"90f5f5b087e676324743bb70e372b49321bf942f63c6ab80892670605eba0c92"} Sep 09 21:07:48.529754 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:48.529681 2562 scope.go:117] "RemoveContainer" containerID="7fe356e7ae497a244d1eb387bf7d2b45a9dc845be3b76829900256533d5f231a" Sep 09 21:07:48.538963 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:48.538925 2562 scope.go:117] "RemoveContainer" containerID="d987779843901d4b17492bf148c79f557a5eb52ccb324bfc40d86865e4a5a450" Sep 09 21:07:48.546458 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:48.546440 2562 scope.go:117] "RemoveContainer" containerID="b82da19f22655fa0c79d4421ccbfd40f286914c3330a7de08acb145097055155" Sep 09 21:07:48.554436 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:48.554404 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk"] Sep 09 21:07:48.555065 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:48.555045 2562 scope.go:117] "RemoveContainer" containerID="7fe356e7ae497a244d1eb387bf7d2b45a9dc845be3b76829900256533d5f231a" Sep 09 21:07:48.555383 ip-10-0-136-126 kubenswrapper[2562]: E0909 21:07:48.555363 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe356e7ae497a244d1eb387bf7d2b45a9dc845be3b76829900256533d5f231a\": container with ID starting with 7fe356e7ae497a244d1eb387bf7d2b45a9dc845be3b76829900256533d5f231a not found: ID does not exist" containerID="7fe356e7ae497a244d1eb387bf7d2b45a9dc845be3b76829900256533d5f231a" Sep 09 21:07:48.555434 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:48.555393 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe356e7ae497a244d1eb387bf7d2b45a9dc845be3b76829900256533d5f231a"} err="failed to get container status \"7fe356e7ae497a244d1eb387bf7d2b45a9dc845be3b76829900256533d5f231a\": rpc error: code = NotFound desc = could not find container \"7fe356e7ae497a244d1eb387bf7d2b45a9dc845be3b76829900256533d5f231a\": container with ID starting with 7fe356e7ae497a244d1eb387bf7d2b45a9dc845be3b76829900256533d5f231a not found: ID does not exist" Sep 09 21:07:48.555434 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:48.555413 2562 scope.go:117] "RemoveContainer" containerID="d987779843901d4b17492bf148c79f557a5eb52ccb324bfc40d86865e4a5a450" Sep 09 21:07:48.555717 ip-10-0-136-126 kubenswrapper[2562]: E0909 21:07:48.555693 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d987779843901d4b17492bf148c79f557a5eb52ccb324bfc40d86865e4a5a450\": container with ID starting with d987779843901d4b17492bf148c79f557a5eb52ccb324bfc40d86865e4a5a450 not found: ID does not exist" containerID="d987779843901d4b17492bf148c79f557a5eb52ccb324bfc40d86865e4a5a450" Sep 09 21:07:48.555772 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:48.555722 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d987779843901d4b17492bf148c79f557a5eb52ccb324bfc40d86865e4a5a450"} err="failed to get container status \"d987779843901d4b17492bf148c79f557a5eb52ccb324bfc40d86865e4a5a450\": rpc error: code = NotFound desc = could not find container \"d987779843901d4b17492bf148c79f557a5eb52ccb324bfc40d86865e4a5a450\": container with ID starting with d987779843901d4b17492bf148c79f557a5eb52ccb324bfc40d86865e4a5a450 not found: ID does not exist" Sep 09 21:07:48.555772 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:48.555740 2562 scope.go:117] "RemoveContainer" containerID="b82da19f22655fa0c79d4421ccbfd40f286914c3330a7de08acb145097055155" Sep 09 21:07:48.556003 ip-10-0-136-126 kubenswrapper[2562]: E0909 21:07:48.555986 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b82da19f22655fa0c79d4421ccbfd40f286914c3330a7de08acb145097055155\": container with ID starting with b82da19f22655fa0c79d4421ccbfd40f286914c3330a7de08acb145097055155 not found: ID does not exist" containerID="b82da19f22655fa0c79d4421ccbfd40f286914c3330a7de08acb145097055155" Sep 09 21:07:48.556051 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:48.556008 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b82da19f22655fa0c79d4421ccbfd40f286914c3330a7de08acb145097055155"} err="failed to get container status \"b82da19f22655fa0c79d4421ccbfd40f286914c3330a7de08acb145097055155\": rpc error: code = NotFound desc = could not find container \"b82da19f22655fa0c79d4421ccbfd40f286914c3330a7de08acb145097055155\": container with ID starting with b82da19f22655fa0c79d4421ccbfd40f286914c3330a7de08acb145097055155 not found: ID does not exist" Sep 09 21:07:48.559639 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:48.559619 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-7dblk"] Sep 09 21:07:49.490654 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:49.490618 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8ffe01c-b06a-4073-9007-08d8fba212bd" path="/var/lib/kubelet/pods/e8ffe01c-b06a-4073-9007-08d8fba212bd/volumes" Sep 09 21:07:50.330802 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:50.330762 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" podUID="2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Sep 09 21:07:58.208077 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:58.208014 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:07:59.584990 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:07:59.584957 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:08:00.331642 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:00.331597 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" podUID="2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Sep 09 21:08:10.331085 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:10.331050 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" Sep 09 21:08:18.216014 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:18.215968 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:08:18.685100 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:18.685066 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:08:28.295718 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:28.295683 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:08:28.432946 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:28.432902 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:08:28.541496 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:28.541452 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:08:28.585453 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:28.585369 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl"] Sep 09 21:08:28.585695 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:28.585669 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" podUID="2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" containerName="kserve-container" containerID="cri-o://0881c823ff0dd46b669d4ab52c5dcfaea0ddffe314aebf368a54ff02b2a5ca1a" gracePeriod=30 Sep 09 21:08:28.585804 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:28.585701 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" podUID="2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" containerName="kube-rbac-proxy" containerID="cri-o://9ed727270fd0da190a4f98b4d1b5f84db79c351e467e13c3661d5c5911790843" gracePeriod=30 Sep 09 21:08:28.631563 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:28.631524 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:08:28.705420 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:28.705383 2562 generic.go:356] "Generic (PLEG): container finished" podID="2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" containerID="9ed727270fd0da190a4f98b4d1b5f84db79c351e467e13c3661d5c5911790843" exitCode=2 Sep 09 21:08:28.705546 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:28.705448 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" event={"ID":"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0","Type":"ContainerDied","Data":"9ed727270fd0da190a4f98b4d1b5f84db79c351e467e13c3661d5c5911790843"} Sep 09 21:08:28.962696 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:28.962662 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:08:30.326286 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:30.326240 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" podUID="2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused" Sep 09 21:08:30.330689 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:30.330655 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" podUID="2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Sep 09 21:08:32.427957 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.427909 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" Sep 09 21:08:32.515712 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.515614 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-cabundle-cert\") pod \"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0\" (UID: \"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0\") " Sep 09 21:08:32.515712 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.515663 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwnp8\" (UniqueName: \"kubernetes.io/projected/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-kube-api-access-mwnp8\") pod \"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0\" (UID: \"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0\") " Sep 09 21:08:32.515712 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.515707 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-kserve-provision-location\") pod \"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0\" (UID: \"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0\") " Sep 09 21:08:32.515989 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.515734 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-proxy-tls\") pod \"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0\" (UID: \"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0\") " Sep 09 21:08:32.515989 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.515777 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-kube-rbac-proxy-sar-config\") pod \"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0\" (UID: \"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0\") " Sep 09 21:08:32.516118 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.516088 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" (UID: "2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 21:08:32.516183 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.516113 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" (UID: "2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 21:08:32.516272 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.516250 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" (UID: "2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 21:08:32.517877 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.517858 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-kube-api-access-mwnp8" (OuterVolumeSpecName: "kube-api-access-mwnp8") pod "2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" (UID: "2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0"). InnerVolumeSpecName "kube-api-access-mwnp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 21:08:32.517877 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.517867 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" (UID: "2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 21:08:32.617164 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.617121 2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-proxy-tls\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 21:08:32.617164 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.617155 2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 21:08:32.617164 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.617166 2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-cabundle-cert\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 21:08:32.617164 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.617176 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwnp8\" (UniqueName: \"kubernetes.io/projected/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-kube-api-access-mwnp8\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 21:08:32.617431 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.617184 2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0-kserve-provision-location\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 21:08:32.723491 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.723459 2562 generic.go:356] "Generic (PLEG): container finished" podID="2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" containerID="0881c823ff0dd46b669d4ab52c5dcfaea0ddffe314aebf368a54ff02b2a5ca1a" exitCode=0 Sep 09 21:08:32.723690 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.723512 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" Sep 09 21:08:32.723690 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.723547 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" event={"ID":"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0","Type":"ContainerDied","Data":"0881c823ff0dd46b669d4ab52c5dcfaea0ddffe314aebf368a54ff02b2a5ca1a"} Sep 09 21:08:32.723690 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.723619 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl" event={"ID":"2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0","Type":"ContainerDied","Data":"92e6d32ce748632325dc57987884b3c312ae9c4afbbf34e6eab1a60c8fb6adb5"} Sep 09 21:08:32.723690 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.723658 2562 scope.go:117] "RemoveContainer" containerID="9ed727270fd0da190a4f98b4d1b5f84db79c351e467e13c3661d5c5911790843" Sep 09 21:08:32.732238 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.732217 2562 scope.go:117] "RemoveContainer" containerID="0881c823ff0dd46b669d4ab52c5dcfaea0ddffe314aebf368a54ff02b2a5ca1a" Sep 09 21:08:32.739714 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.739689 2562 scope.go:117] "RemoveContainer" containerID="a54ed759771062b0f1c792f9eb7d20e5a1a1db0ed53d3a965444d8051ae2c9c9" Sep 09 21:08:32.745849 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.745826 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl"] Sep 09 21:08:32.747771 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.747751 2562 scope.go:117] "RemoveContainer" containerID="9ed727270fd0da190a4f98b4d1b5f84db79c351e467e13c3661d5c5911790843" Sep 09 21:08:32.748072 ip-10-0-136-126 kubenswrapper[2562]: E0909 21:08:32.748054 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ed727270fd0da190a4f98b4d1b5f84db79c351e467e13c3661d5c5911790843\": container with ID starting with 9ed727270fd0da190a4f98b4d1b5f84db79c351e467e13c3661d5c5911790843 not found: ID does not exist" containerID="9ed727270fd0da190a4f98b4d1b5f84db79c351e467e13c3661d5c5911790843" Sep 09 21:08:32.748126 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.748082 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed727270fd0da190a4f98b4d1b5f84db79c351e467e13c3661d5c5911790843"} err="failed to get container status \"9ed727270fd0da190a4f98b4d1b5f84db79c351e467e13c3661d5c5911790843\": rpc error: code = NotFound desc = could not find container \"9ed727270fd0da190a4f98b4d1b5f84db79c351e467e13c3661d5c5911790843\": container with ID starting with 9ed727270fd0da190a4f98b4d1b5f84db79c351e467e13c3661d5c5911790843 not found: ID does not exist" Sep 09 21:08:32.748126 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.748101 2562 scope.go:117] "RemoveContainer" containerID="0881c823ff0dd46b669d4ab52c5dcfaea0ddffe314aebf368a54ff02b2a5ca1a" Sep 09 21:08:32.748357 ip-10-0-136-126 kubenswrapper[2562]: E0909 21:08:32.748337 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0881c823ff0dd46b669d4ab52c5dcfaea0ddffe314aebf368a54ff02b2a5ca1a\": container with ID starting with 0881c823ff0dd46b669d4ab52c5dcfaea0ddffe314aebf368a54ff02b2a5ca1a not found: ID does not exist" containerID="0881c823ff0dd46b669d4ab52c5dcfaea0ddffe314aebf368a54ff02b2a5ca1a" Sep 09 21:08:32.748396 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.748367 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0881c823ff0dd46b669d4ab52c5dcfaea0ddffe314aebf368a54ff02b2a5ca1a"} err="failed to get container status \"0881c823ff0dd46b669d4ab52c5dcfaea0ddffe314aebf368a54ff02b2a5ca1a\": rpc error: code = NotFound desc = could not find container \"0881c823ff0dd46b669d4ab52c5dcfaea0ddffe314aebf368a54ff02b2a5ca1a\": container with ID starting with 0881c823ff0dd46b669d4ab52c5dcfaea0ddffe314aebf368a54ff02b2a5ca1a not found: ID does not exist" Sep 09 21:08:32.748396 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.748385 2562 scope.go:117] "RemoveContainer" containerID="a54ed759771062b0f1c792f9eb7d20e5a1a1db0ed53d3a965444d8051ae2c9c9" Sep 09 21:08:32.748603 ip-10-0-136-126 kubenswrapper[2562]: E0909 21:08:32.748587 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a54ed759771062b0f1c792f9eb7d20e5a1a1db0ed53d3a965444d8051ae2c9c9\": container with ID starting with a54ed759771062b0f1c792f9eb7d20e5a1a1db0ed53d3a965444d8051ae2c9c9 not found: ID does not exist" containerID="a54ed759771062b0f1c792f9eb7d20e5a1a1db0ed53d3a965444d8051ae2c9c9" Sep 09 21:08:32.748647 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.748607 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54ed759771062b0f1c792f9eb7d20e5a1a1db0ed53d3a965444d8051ae2c9c9"} err="failed to get container status \"a54ed759771062b0f1c792f9eb7d20e5a1a1db0ed53d3a965444d8051ae2c9c9\": rpc error: code = NotFound desc = could not find container \"a54ed759771062b0f1c792f9eb7d20e5a1a1db0ed53d3a965444d8051ae2c9c9\": container with ID starting with a54ed759771062b0f1c792f9eb7d20e5a1a1db0ed53d3a965444d8051ae2c9c9 not found: ID does not exist" Sep 09 21:08:32.752418 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:32.752396 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-887b3-predictor-54f6b57485-jzrsl"] Sep 09 21:08:33.489834 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:33.489798 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" path="/var/lib/kubelet/pods/2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0/volumes" Sep 09 21:08:48.493276 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:48.493237 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:08:48.795818 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:08:48.795737 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:09:43.359069 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:09:43.359033 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:09:43.517887 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:09:43.517835 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:10:01.486902 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:10:01.486859 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:10:01.573555 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:10:01.573522 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:10:06.948038 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:10:06.947995 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:10:07.016171 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:10:07.016135 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:10:16.699530 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:10:16.699447 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:10:21.959768 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:10:21.959718 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:10:22.246521 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:10:22.246424 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:10:23.747708 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:10:23.747661 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:10:23.874264 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:10:23.874230 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:10:23.973806 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:10:23.973773 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:10:24.042824 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:10:24.042742 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:11:52.050366 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:11:52.050331 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:11:53.118128 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:11:53.118090 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:11:57.204073 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:11:57.204041 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:11:57.352111 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:11:57.352077 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:12:02.114519 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:12:02.114484 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:12:19.447562 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:12:19.447536 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-qlzzw_247631db-a55e-4fbc-9bde-f921c2bb309c/console-operator/1.log" Sep 09 21:12:19.449604 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:12:19.449580 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-qlzzw_247631db-a55e-4fbc-9bde-f921c2bb309c/console-operator/1.log" Sep 09 21:12:19.454142 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:12:19.454119 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crjwd_d55a4588-7960-4998-a99e-fca8f803b053/ovn-acl-logging/0.log" Sep 09 21:12:19.455814 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:12:19.455796 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crjwd_d55a4588-7960-4998-a99e-fca8f803b053/ovn-acl-logging/0.log" Sep 09 21:12:34.528326 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:12:34.528287 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:13:13.946038 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:13:13.945901 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:13:19.069904 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:13:19.069862 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:13:19.158242 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:13:19.158205 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:13:19.316755 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:13:19.316717 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:13:19.416227 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:13:19.416191 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:13:27.336355 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:13:27.336316 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:14:27.164477 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:14:27.164438 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:14:49.245680 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:14:49.245643 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:14:49.324275 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:14:49.324237 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:14:54.316517 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:14:54.316482 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:14:54.415985 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:14:54.415943 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:14:54.568283 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:14:54.568199 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:14:54.781570 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:14:54.781515 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:16:24.464986 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:16:24.464872 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:16:59.562987 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:16:59.562954 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:17:19.472926 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:17:19.472894 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-qlzzw_247631db-a55e-4fbc-9bde-f921c2bb309c/console-operator/1.log" Sep 09 21:17:19.475891 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:17:19.475867 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-qlzzw_247631db-a55e-4fbc-9bde-f921c2bb309c/console-operator/1.log" Sep 09 21:17:19.479551 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:17:19.479531 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crjwd_d55a4588-7960-4998-a99e-fca8f803b053/ovn-acl-logging/0.log" Sep 09 21:17:19.482689 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:17:19.482671 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-crjwd_d55a4588-7960-4998-a99e-fca8f803b053/ovn-acl-logging/0.log" Sep 09 21:17:55.748010 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:17:55.747978 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:18:09.132210 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:18:09.132179 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:18:09.363113 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:18:09.363074 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:18:19.238691 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:18:19.238650 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:18:20.773245 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:18:20.773212 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:18:34.454382 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:18:34.454343 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:19:54.527445 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:54.527400 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:19:54.591531 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:54.591501 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:19:55.885286 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.885243 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:19:55.889583 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.889554 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j9fnj/must-gather-9g8mn"] Sep 09 21:19:55.889854 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.889842 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" containerName="kserve-container" Sep 09 21:19:55.889896 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.889856 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" containerName="kserve-container" Sep 09 21:19:55.889896 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.889869 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ffe01c-b06a-4073-9007-08d8fba212bd" containerName="kube-rbac-proxy" Sep 09 21:19:55.889896 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.889876 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ffe01c-b06a-4073-9007-08d8fba212bd" containerName="kube-rbac-proxy" Sep 09 21:19:55.889896 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.889885 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ffe01c-b06a-4073-9007-08d8fba212bd" containerName="kserve-container" Sep 09 21:19:55.889896 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.889891 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ffe01c-b06a-4073-9007-08d8fba212bd" containerName="kserve-container" Sep 09 21:19:55.890087 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.889899 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ffe01c-b06a-4073-9007-08d8fba212bd" containerName="storage-initializer" Sep 09 21:19:55.890087 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.889904 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ffe01c-b06a-4073-9007-08d8fba212bd" containerName="storage-initializer" Sep 09 21:19:55.890087 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.889915 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" containerName="kube-rbac-proxy" Sep 09 21:19:55.890087 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.889920 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" containerName="kube-rbac-proxy" Sep 09 21:19:55.890087 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.889941 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" containerName="storage-initializer" Sep 09 21:19:55.890087 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.889950 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" containerName="storage-initializer" Sep 09 21:19:55.890087 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.890015 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8ffe01c-b06a-4073-9007-08d8fba212bd" containerName="kserve-container" Sep 09 21:19:55.890087 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.890023 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" containerName="kserve-container" Sep 09 21:19:55.890087 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.890032 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8ffe01c-b06a-4073-9007-08d8fba212bd" containerName="kube-rbac-proxy" Sep 09 21:19:55.890087 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.890041 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e9870a4-53b7-4ad9-b3ed-49fb473fa8c0" containerName="kube-rbac-proxy" Sep 09 21:19:55.893072 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.893056 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j9fnj/must-gather-9g8mn" Sep 09 21:19:55.895794 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.895772 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j9fnj\"/\"openshift-service-ca.crt\"" Sep 09 21:19:55.896625 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.896609 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j9fnj\"/\"default-dockercfg-qqw5c\"" Sep 09 21:19:55.896684 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.896663 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j9fnj\"/\"kube-root-ca.crt\"" Sep 09 21:19:55.899555 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.899530 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j9fnj/must-gather-9g8mn"] Sep 09 21:19:55.957263 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.957226 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:19:55.972272 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.972234 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2bkg\" (UniqueName: \"kubernetes.io/projected/261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf-kube-api-access-g2bkg\") pod \"must-gather-9g8mn\" (UID: \"261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf\") " pod="openshift-must-gather-j9fnj/must-gather-9g8mn" Sep 09 21:19:55.972388 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:55.972292 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf-must-gather-output\") pod \"must-gather-9g8mn\" (UID: \"261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf\") " pod="openshift-must-gather-j9fnj/must-gather-9g8mn" Sep 09 21:19:56.073595 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:56.073544 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2bkg\" (UniqueName: \"kubernetes.io/projected/261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf-kube-api-access-g2bkg\") pod \"must-gather-9g8mn\" (UID: \"261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf\") " pod="openshift-must-gather-j9fnj/must-gather-9g8mn" Sep 09 21:19:56.073790 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:56.073679 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf-must-gather-output\") pod \"must-gather-9g8mn\" (UID: \"261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf\") " pod="openshift-must-gather-j9fnj/must-gather-9g8mn" Sep 09 21:19:56.074045 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:56.074029 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf-must-gather-output\") pod \"must-gather-9g8mn\" (UID: \"261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf\") " pod="openshift-must-gather-j9fnj/must-gather-9g8mn" Sep 09 21:19:56.083924 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:56.083892 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2bkg\" (UniqueName: \"kubernetes.io/projected/261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf-kube-api-access-g2bkg\") pod \"must-gather-9g8mn\" (UID: \"261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf\") " pod="openshift-must-gather-j9fnj/must-gather-9g8mn" Sep 09 21:19:56.215798 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:56.215702 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j9fnj/must-gather-9g8mn" Sep 09 21:19:56.487348 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:56.487322 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j9fnj/must-gather-9g8mn"] Sep 09 21:19:56.489439 ip-10-0-136-126 kubenswrapper[2562]: W0909 21:19:56.489411 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod261cdd25_cfc9_443f_b4ed_b3e2d3cee5cf.slice/crio-50762168d570a604809e5374a2e5ff4e4cc125c5a5509eae22130a71c0e55b6d WatchSource:0}: Error finding container 50762168d570a604809e5374a2e5ff4e4cc125c5a5509eae22130a71c0e55b6d: Status 404 returned error can't find the container with id 50762168d570a604809e5374a2e5ff4e4cc125c5a5509eae22130a71c0e55b6d Sep 09 21:19:56.491181 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:56.491163 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 21:19:57.465160 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:19:57.465117 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j9fnj/must-gather-9g8mn" event={"ID":"261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf","Type":"ContainerStarted","Data":"50762168d570a604809e5374a2e5ff4e4cc125c5a5509eae22130a71c0e55b6d"} Sep 09 21:20:03.495467 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:03.495421 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j9fnj/must-gather-9g8mn" event={"ID":"261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf","Type":"ContainerStarted","Data":"212bbeef22f2e8075193a2b5117b2a69100d76009ecbb3eb3cfd1af13e6d40d8"} Sep 09 21:20:03.495467 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:03.495474 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j9fnj/must-gather-9g8mn" event={"ID":"261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf","Type":"ContainerStarted","Data":"2c2b560c4663d0968b476b84299dcdc39c3a94df24f1f9ba46ffa2e48cd7a45c"} Sep 09 21:20:03.514618 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:03.514554 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j9fnj/must-gather-9g8mn" podStartSLOduration=2.466677819 podStartE2EDuration="8.51453864s" totalImagesPullingTime="6.047860821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:20:03.514373196 +0000 UTC m=+1664.556373010" watchObservedRunningTime="2026-09-09 21:20:03.51453864 +0000 UTC m=+1664.556538449" Sep 09 21:20:26.598580 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:26.598540 2562 generic.go:356] "Generic (PLEG): container finished" podID="261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf" containerID="2c2b560c4663d0968b476b84299dcdc39c3a94df24f1f9ba46ffa2e48cd7a45c" exitCode=0 Sep 09 21:20:26.599050 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:26.598612 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j9fnj/must-gather-9g8mn" event={"ID":"261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf","Type":"ContainerDied","Data":"2c2b560c4663d0968b476b84299dcdc39c3a94df24f1f9ba46ffa2e48cd7a45c"} Sep 09 21:20:26.599050 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:26.598954 2562 scope.go:117] "RemoveContainer" containerID="2c2b560c4663d0968b476b84299dcdc39c3a94df24f1f9ba46ffa2e48cd7a45c" Sep 09 21:20:26.640672 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:26.640643 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-j9fnj_must-gather-9g8mn_261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf/gather/0.log" Sep 09 21:20:27.078162 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:27.078070 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:20:27.260670 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:27.260626 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:20:27.358419 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:27.358386 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:20:31.156854 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:31.156823 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kf998_0b2c736d-7746-4285-93bc-5ffd114396f1/global-pull-secret-syncer/0.log" Sep 09 21:20:31.210988 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:31.210959 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" Sep 09 21:20:31.326352 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:31.326323 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-126.ec2.internal_63346b7c1fd6eaae6b1974fddafa03ee/haproxy/0.log" Sep 09 21:20:32.052485 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:32.052450 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-j9fnj/must-gather-9g8mn"] Sep 09 21:20:32.052680 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:32.052641 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-j9fnj/must-gather-9g8mn" podUID="261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf" containerName="copy" containerID="cri-o://212bbeef22f2e8075193a2b5117b2a69100d76009ecbb3eb3cfd1af13e6d40d8" gracePeriod=2 Sep 09 21:20:32.059663 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:32.059632 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-j9fnj/must-gather-9g8mn"] Sep 09 21:20:32.283502 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:32.283474 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-j9fnj_must-gather-9g8mn_261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf/copy/0.log" Sep 09 21:20:32.285568 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:32.285548 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j9fnj/must-gather-9g8mn" Sep 09 21:20:32.287809 ip-10-0-136-126 kubenswrapper[2562]: E0909 21:20:32.287783 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-9g8mn\" is forbidden: User \"system:node:ip-10-0-136-126.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-j9fnj\": no relationship found between node 'ip-10-0-136-126.ec2.internal' and this object" podUID="261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf" pod="openshift-must-gather-j9fnj/must-gather-9g8mn" Sep 09 21:20:32.303280 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:32.303198 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf-must-gather-output\") pod \"261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf\" (UID: \"261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf\") " Sep 09 21:20:32.306177 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:32.306148 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf" (UID: "261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 21:20:32.403672 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:32.403637 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2bkg\" (UniqueName: \"kubernetes.io/projected/261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf-kube-api-access-g2bkg\") pod \"261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf\" (UID: \"261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf\") " Sep 09 21:20:32.403846 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:32.403763 2562 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf-must-gather-output\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 21:20:32.405866 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:32.405836 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf-kube-api-access-g2bkg" (OuterVolumeSpecName: "kube-api-access-g2bkg") pod "261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf" (UID: "261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf"). InnerVolumeSpecName "kube-api-access-g2bkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 21:20:32.504532 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:32.504487 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2bkg\" (UniqueName: \"kubernetes.io/projected/261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf-kube-api-access-g2bkg\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Sep 09 21:20:32.624530 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:32.624503 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-j9fnj_must-gather-9g8mn_261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf/copy/0.log" Sep 09 21:20:32.624972 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:32.624947 2562 generic.go:356] "Generic (PLEG): container finished" podID="261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf" containerID="212bbeef22f2e8075193a2b5117b2a69100d76009ecbb3eb3cfd1af13e6d40d8" exitCode=143 Sep 09 21:20:32.625070 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:32.625002 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j9fnj/must-gather-9g8mn" Sep 09 21:20:32.625070 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:32.625045 2562 scope.go:117] "RemoveContainer" containerID="212bbeef22f2e8075193a2b5117b2a69100d76009ecbb3eb3cfd1af13e6d40d8" Sep 09 21:20:32.626916 ip-10-0-136-126 kubenswrapper[2562]: E0909 21:20:32.626893 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-9g8mn\" is forbidden: User \"system:node:ip-10-0-136-126.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-j9fnj\": no relationship found between node 'ip-10-0-136-126.ec2.internal' and this object" podUID="261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf" pod="openshift-must-gather-j9fnj/must-gather-9g8mn" Sep 09 21:20:32.633536 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:32.633512 2562 scope.go:117] "RemoveContainer" containerID="2c2b560c4663d0968b476b84299dcdc39c3a94df24f1f9ba46ffa2e48cd7a45c" Sep 09 21:20:32.635781 ip-10-0-136-126 kubenswrapper[2562]: E0909 21:20:32.635758 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-9g8mn\" is forbidden: User \"system:node:ip-10-0-136-126.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-j9fnj\": no relationship found between node 'ip-10-0-136-126.ec2.internal' and this object" podUID="261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf" pod="openshift-must-gather-j9fnj/must-gather-9g8mn" Sep 09 21:20:32.646288 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:32.646268 2562 scope.go:117] "RemoveContainer" containerID="212bbeef22f2e8075193a2b5117b2a69100d76009ecbb3eb3cfd1af13e6d40d8" Sep 09 21:20:32.646579 ip-10-0-136-126 kubenswrapper[2562]: E0909 21:20:32.646562 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"212bbeef22f2e8075193a2b5117b2a69100d76009ecbb3eb3cfd1af13e6d40d8\": container with ID starting with 212bbeef22f2e8075193a2b5117b2a69100d76009ecbb3eb3cfd1af13e6d40d8 not found: ID does not exist" containerID="212bbeef22f2e8075193a2b5117b2a69100d76009ecbb3eb3cfd1af13e6d40d8" Sep 09 21:20:32.646625 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:32.646589 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"212bbeef22f2e8075193a2b5117b2a69100d76009ecbb3eb3cfd1af13e6d40d8"} err="failed to get container status \"212bbeef22f2e8075193a2b5117b2a69100d76009ecbb3eb3cfd1af13e6d40d8\": rpc error: code = NotFound desc = could not find container \"212bbeef22f2e8075193a2b5117b2a69100d76009ecbb3eb3cfd1af13e6d40d8\": container with ID starting with 212bbeef22f2e8075193a2b5117b2a69100d76009ecbb3eb3cfd1af13e6d40d8 not found: ID does not exist" Sep 09 21:20:32.646625 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:32.646610 2562 scope.go:117] "RemoveContainer" containerID="2c2b560c4663d0968b476b84299dcdc39c3a94df24f1f9ba46ffa2e48cd7a45c" Sep 09 21:20:32.646813 ip-10-0-136-126 kubenswrapper[2562]: E0909 21:20:32.646792 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c2b560c4663d0968b476b84299dcdc39c3a94df24f1f9ba46ffa2e48cd7a45c\": container with ID starting with 2c2b560c4663d0968b476b84299dcdc39c3a94df24f1f9ba46ffa2e48cd7a45c not found: ID does not exist" containerID="2c2b560c4663d0968b476b84299dcdc39c3a94df24f1f9ba46ffa2e48cd7a45c" Sep 09 21:20:32.646862 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:32.646824 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c2b560c4663d0968b476b84299dcdc39c3a94df24f1f9ba46ffa2e48cd7a45c"} err="failed to get container status \"2c2b560c4663d0968b476b84299dcdc39c3a94df24f1f9ba46ffa2e48cd7a45c\": rpc error: code = NotFound desc = could not find container \"2c2b560c4663d0968b476b84299dcdc39c3a94df24f1f9ba46ffa2e48cd7a45c\": container with ID starting with 2c2b560c4663d0968b476b84299dcdc39c3a94df24f1f9ba46ffa2e48cd7a45c not found: ID does not exist" Sep 09 21:20:33.027075 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:33.026960 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:20:33.123827 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:33.123797 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gvw72_a88d1a84-874b-464b-b9c7-8952100b363e/konnectivity-agent/0.log" limit=1024 Sep 09 21:20:33.489561 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:33.489529 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf" path="/var/lib/kubelet/pods/261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf/volumes" Sep 09 21:20:36.051700 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:36.051628 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-b5v84_0ca1e078-17b2-4050-ba7b-202df948a805/kube-state-metrics/0.log" Sep 09 21:20:36.081221 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:36.081181 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-b5v84_0ca1e078-17b2-4050-ba7b-202df948a805/kube-rbac-proxy-main/0.log" Sep 09 21:20:36.105633 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:36.105592 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-b5v84_0ca1e078-17b2-4050-ba7b-202df948a805/kube-rbac-proxy-self/0.log" Sep 09 21:20:36.147093 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:36.147062 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-646bcb899d-882pt_d9ef4dc8-92d8-4e62-8b3e-69b3c3e2f708/metrics-server/0.log" Sep 09 21:20:36.183774 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:36.183747 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-fns2c_1b087158-63bd-4adc-a13f-0fee6d559f51/monitoring-plugin/0.log" Sep 09 21:20:36.309460 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:36.309377 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lptbr_a8b7eabc-e75b-436f-be68-413fe8ba86d0/node-exporter/0.log" Sep 09 21:20:36.338441 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:36.338411 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lptbr_a8b7eabc-e75b-436f-be68-413fe8ba86d0/kube-rbac-proxy/0.log" Sep 09 21:20:36.362906 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:36.362879 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lptbr_a8b7eabc-e75b-436f-be68-413fe8ba86d0/init-textfile/0.log" Sep 09 21:20:36.593759 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:36.593716 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b192551-2069-4ef6-a077-f6f22c73d5a3/prometheus/0.log" Sep 09 21:20:36.619263 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:36.619229 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b192551-2069-4ef6-a077-f6f22c73d5a3/config-reloader/0.log" Sep 09 21:20:36.646393 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:36.646358 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b192551-2069-4ef6-a077-f6f22c73d5a3/thanos-sidecar/0.log" Sep 09 21:20:36.675490 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:36.675468 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b192551-2069-4ef6-a077-f6f22c73d5a3/kube-rbac-proxy-web/0.log" Sep 09 21:20:36.722669 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:36.722640 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b192551-2069-4ef6-a077-f6f22c73d5a3/kube-rbac-proxy/0.log" Sep 09 21:20:36.769736 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:36.769708 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b192551-2069-4ef6-a077-f6f22c73d5a3/kube-rbac-proxy-thanos/0.log" Sep 09 21:20:36.816693 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:36.816668 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b192551-2069-4ef6-a077-f6f22c73d5a3/init-config-reloader/0.log" Sep 09 21:20:36.862639 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:36.862556 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-p942p_20d7ee39-f1be-4d32-9f91-9015bc14779f/prometheus-operator/0.log" Sep 09 21:20:36.925076 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:36.925047 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-p942p_20d7ee39-f1be-4d32-9f91-9015bc14779f/kube-rbac-proxy/0.log" Sep 09 21:20:36.967302 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:36.967266 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-bk2tb_ec2d4891-a6a2-4729-8ff8-f733dd672361/prometheus-operator-admission-webhook/0.log" Sep 09 21:20:38.516525 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:38.516488 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-cljj5_8d11438c-9ef0-491b-9813-36b419613f92/networking-console-plugin/0.log" Sep 09 21:20:39.074514 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.074478 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-qlzzw_247631db-a55e-4fbc-9bde-f921c2bb309c/console-operator/1.log" Sep 09 21:20:39.079505 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.079481 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-qlzzw_247631db-a55e-4fbc-9bde-f921c2bb309c/console-operator/2.log" Sep 09 21:20:39.166112 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.166077 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wltvn/perf-node-gather-daemonset-qsr4p"] Sep 09 21:20:39.166481 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.166463 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf" containerName="gather" Sep 09 21:20:39.166572 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.166483 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf" containerName="gather" Sep 09 21:20:39.166572 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.166497 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf" containerName="copy" Sep 09 21:20:39.166572 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.166504 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf" containerName="copy" Sep 09 21:20:39.166717 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.166594 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf" containerName="copy" Sep 09 21:20:39.166717 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.166611 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="261cdd25-cfc9-443f-b4ed-b3e2d3cee5cf" containerName="gather" Sep 09 21:20:39.170758 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.170735 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-qsr4p" Sep 09 21:20:39.173452 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.173428 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wltvn\"/\"openshift-service-ca.crt\"" Sep 09 21:20:39.173590 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.173428 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wltvn\"/\"default-dockercfg-rjrx7\"" Sep 09 21:20:39.174109 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.174094 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wltvn\"/\"kube-root-ca.crt\"" Sep 09 21:20:39.184017 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.183992 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wltvn/perf-node-gather-daemonset-qsr4p"] Sep 09 21:20:39.258685 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.258646 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7479261c-1be8-4886-9a31-6e01db1f0927-podres\") pod \"perf-node-gather-daemonset-qsr4p\" (UID: \"7479261c-1be8-4886-9a31-6e01db1f0927\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-qsr4p" Sep 09 21:20:39.258685 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.258696 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7479261c-1be8-4886-9a31-6e01db1f0927-sys\") pod \"perf-node-gather-daemonset-qsr4p\" (UID: \"7479261c-1be8-4886-9a31-6e01db1f0927\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-qsr4p" Sep 09 21:20:39.258922 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.258712 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7479261c-1be8-4886-9a31-6e01db1f0927-lib-modules\") pod \"perf-node-gather-daemonset-qsr4p\" (UID: \"7479261c-1be8-4886-9a31-6e01db1f0927\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-qsr4p" Sep 09 21:20:39.258922 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.258799 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hj8s\" (UniqueName: \"kubernetes.io/projected/7479261c-1be8-4886-9a31-6e01db1f0927-kube-api-access-9hj8s\") pod \"perf-node-gather-daemonset-qsr4p\" (UID: \"7479261c-1be8-4886-9a31-6e01db1f0927\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-qsr4p" Sep 09 21:20:39.258922 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.258840 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7479261c-1be8-4886-9a31-6e01db1f0927-proc\") pod \"perf-node-gather-daemonset-qsr4p\" (UID: \"7479261c-1be8-4886-9a31-6e01db1f0927\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-qsr4p" Sep 09 21:20:39.359665 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.359626 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7479261c-1be8-4886-9a31-6e01db1f0927-podres\") pod \"perf-node-gather-daemonset-qsr4p\" (UID: \"7479261c-1be8-4886-9a31-6e01db1f0927\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-qsr4p" Sep 09 21:20:39.359844 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.359687 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7479261c-1be8-4886-9a31-6e01db1f0927-sys\") pod \"perf-node-gather-daemonset-qsr4p\" (UID: \"7479261c-1be8-4886-9a31-6e01db1f0927\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-qsr4p" Sep 09 21:20:39.359844 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.359712 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7479261c-1be8-4886-9a31-6e01db1f0927-lib-modules\") pod \"perf-node-gather-daemonset-qsr4p\" (UID: \"7479261c-1be8-4886-9a31-6e01db1f0927\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-qsr4p" Sep 09 21:20:39.359844 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.359747 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hj8s\" (UniqueName: \"kubernetes.io/projected/7479261c-1be8-4886-9a31-6e01db1f0927-kube-api-access-9hj8s\") pod \"perf-node-gather-daemonset-qsr4p\" (UID: \"7479261c-1be8-4886-9a31-6e01db1f0927\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-qsr4p" Sep 09 21:20:39.359844 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.359822 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7479261c-1be8-4886-9a31-6e01db1f0927-proc\") pod \"perf-node-gather-daemonset-qsr4p\" (UID: \"7479261c-1be8-4886-9a31-6e01db1f0927\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-qsr4p" Sep 09 21:20:39.360062 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.359849 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7479261c-1be8-4886-9a31-6e01db1f0927-lib-modules\") pod \"perf-node-gather-daemonset-qsr4p\" (UID: \"7479261c-1be8-4886-9a31-6e01db1f0927\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-qsr4p" Sep 09 21:20:39.360062 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.359826 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7479261c-1be8-4886-9a31-6e01db1f0927-podres\") pod \"perf-node-gather-daemonset-qsr4p\" (UID: \"7479261c-1be8-4886-9a31-6e01db1f0927\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-qsr4p" Sep 09 21:20:39.360062 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.359867 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7479261c-1be8-4886-9a31-6e01db1f0927-proc\") pod \"perf-node-gather-daemonset-qsr4p\" (UID: \"7479261c-1be8-4886-9a31-6e01db1f0927\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-qsr4p" Sep 09 21:20:39.360062 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.359829 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7479261c-1be8-4886-9a31-6e01db1f0927-sys\") pod \"perf-node-gather-daemonset-qsr4p\" (UID: \"7479261c-1be8-4886-9a31-6e01db1f0927\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-qsr4p" Sep 09 21:20:39.374986 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.374956 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hj8s\" (UniqueName: \"kubernetes.io/projected/7479261c-1be8-4886-9a31-6e01db1f0927-kube-api-access-9hj8s\") pod \"perf-node-gather-daemonset-qsr4p\" (UID: \"7479261c-1be8-4886-9a31-6e01db1f0927\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-qsr4p" Sep 09 21:20:39.480556 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.480518 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-qsr4p" Sep 09 21:20:39.757475 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.757452 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wltvn/perf-node-gather-daemonset-qsr4p"] Sep 09 21:20:39.759237 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:39.759213 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-jrfbb_1ee25625-54d3-4ac1-912c-42f78f271480/download-server/0.log" Sep 09 21:20:39.759566 ip-10-0-136-126 kubenswrapper[2562]: W0909 21:20:39.759542 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7479261c_1be8_4886_9a31_6e01db1f0927.slice/crio-7d23991a08c5d901d85377fd1285a7d161d5bf8bfaa7a450903b0cb392465823 WatchSource:0}: Error finding container 7d23991a08c5d901d85377fd1285a7d161d5bf8bfaa7a450903b0cb392465823: Status 404 returned error can't find the container with id 7d23991a08c5d901d85377fd1285a7d161d5bf8bfaa7a450903b0cb392465823 Sep 09 21:20:40.660669 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:40.660630 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-qsr4p" event={"ID":"7479261c-1be8-4886-9a31-6e01db1f0927","Type":"ContainerStarted","Data":"24836909aefecff4f71744604da6e56ac4c6f4196d4474367a9a9da2f752e981"} Sep 09 21:20:40.660861 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:40.660716 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-qsr4p" event={"ID":"7479261c-1be8-4886-9a31-6e01db1f0927","Type":"ContainerStarted","Data":"7d23991a08c5d901d85377fd1285a7d161d5bf8bfaa7a450903b0cb392465823"} Sep 09 21:20:40.660861 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:40.660738 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-qsr4p" Sep 09 21:20:40.678171 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:40.678119 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-qsr4p" podStartSLOduration=1.6781012990000002 podStartE2EDuration="1.678101299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:20:40.677076195 +0000 UTC m=+1701.719076030" watchObservedRunningTime="2026-09-09 21:20:40.678101299 +0000 UTC m=+1701.720101108" Sep 09 21:20:41.466456 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:41.466429 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hrmkf_92eda8aa-9402-4df8-b58e-5468b16a888e/dns/0.log" Sep 09 21:20:41.507529 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:41.507499 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hrmkf_92eda8aa-9402-4df8-b58e-5468b16a888e/kube-rbac-proxy/0.log" Sep 09 21:20:41.658483 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:41.658453 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n7kn4_7c46d28d-4f9c-4307-827a-9c9a31213cb2/dns-node-resolver/0.log" Sep 09 21:20:42.145617 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:42.145589 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5cx59_4e4565ab-4f66-4753-b0d0-7d088a90d3da/node-ca/0.log" Sep 09 21:20:43.030204 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:43.030170 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f7bbb8cc6-ptg74_95fc21f9-62ed-41e4-9609-28ffab20ab40/router/0.log" Sep 09 21:20:43.383795 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:43.383770 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c5n7q_bd6d00ed-c027-4de3-962d-edd04fbf3bd5/serve-healthcheck-canary/0.log" Sep 09 21:20:44.048121 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:44.048092 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kfg2r_d9680a3c-cfee-45a8-8793-8a5d5888935b/kube-rbac-proxy/0.log" Sep 09 21:20:44.075808 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:44.075781 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kfg2r_d9680a3c-cfee-45a8-8793-8a5d5888935b/exporter/0.log" Sep 09 21:20:44.102491 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:44.102466 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kfg2r_d9680a3c-cfee-45a8-8793-8a5d5888935b/extractor/0.log" Sep 09 21:20:46.675106 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:46.675076 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-qsr4p" Sep 09 21:20:46.686321 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:46.686298 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-qxvhf_53b7c008-6ea4-43f1-a083-29e6480cebd4/s3-init/0.log" Sep 09 21:20:46.710737 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:46.710704 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-qxvhf_53b7c008-6ea4-43f1-a083-29e6480cebd4/download-hf-model/0.log" Sep 09 21:20:53.810245 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:53.810208 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-69jvv_346ba7a6-b7db-48ac-9dd8-5793933dd7cd/kube-multus/0.log" Sep 09 21:20:54.099843 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:54.099813 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nptv6_d2f2d9c9-c216-491f-b3e9-dac3643cd6db/kube-multus-additional-cni-plugins/0.log" Sep 09 21:20:54.135029 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:54.135001 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nptv6_d2f2d9c9-c216-491f-b3e9-dac3643cd6db/egress-router-binary-copy/0.log" Sep 09 21:20:54.166830 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:54.166803 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nptv6_d2f2d9c9-c216-491f-b3e9-dac3643cd6db/cni-plugins/0.log" Sep 09 21:20:54.196390 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:54.196351 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nptv6_d2f2d9c9-c216-491f-b3e9-dac3643cd6db/bond-cni-plugin/0.log" Sep 09 21:20:54.235310 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:54.235275 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nptv6_d2f2d9c9-c216-491f-b3e9-dac3643cd6db/routeoverride-cni/0.log" Sep 09 21:20:54.266028 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:54.265996 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nptv6_d2f2d9c9-c216-491f-b3e9-dac3643cd6db/whereabouts-cni-bincopy/0.log" Sep 09 21:20:54.300362 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:54.300330 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nptv6_d2f2d9c9-c216-491f-b3e9-dac3643cd6db/whereabouts-cni/0.log" Sep 09 21:20:54.751758 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:54.751720 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-d6q8h_ecd60cff-708d-4a89-827e-e01025163b5a/network-metrics-daemon/0.log" Sep 09 21:20:54.784833 ip-10-0-136-126 kubenswrapper[2562]: I0909 21:20:54.784804 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-d6q8h_ecd60cff-708d-4a89-827e-e01025163b5a/kube-rbac-proxy/0.log"