Sep 09 23:06:51.202189 ip-10-0-143-192 systemd[1]: Starting Kubernetes Kubelet... Sep 09 23:06:51.586461 ip-10-0-143-192 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 23:06:51.586461 ip-10-0-143-192 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 09 23:06:51.586461 ip-10-0-143-192 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 23:06:51.586461 ip-10-0-143-192 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 09 23:06:51.586461 ip-10-0-143-192 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 23:06:51.588262 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.588165 2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 09 23:06:51.591726 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591708 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 23:06:51.591726 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591727 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 23:06:51.591799 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591731 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 23:06:51.591799 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591736 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 23:06:51.591799 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591742 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 23:06:51.591799 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591745 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 23:06:51.591799 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591748 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 23:06:51.591799 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591750 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 23:06:51.591799 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591753 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 23:06:51.591799 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591756 2561 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 23:06:51.591799 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591758 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 23:06:51.591799 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591761 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 23:06:51.591799 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591764 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 23:06:51.591799 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591766 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 23:06:51.591799 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591769 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 23:06:51.591799 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591771 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 23:06:51.591799 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591774 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 23:06:51.591799 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591778 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 23:06:51.591799 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591783 2561 feature_gate.go:328] unrecognized feature gate: Example Sep 09 23:06:51.591799 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591786 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 23:06:51.591799 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591789 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 23:06:51.591799 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591793 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 23:06:51.592294 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591795 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 23:06:51.592294 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591799 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 23:06:51.592294 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591801 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 23:06:51.592294 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591804 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 23:06:51.592294 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591807 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 23:06:51.592294 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591811 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 23:06:51.592294 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591813 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 23:06:51.592294 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591816 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 23:06:51.592294 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591819 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 23:06:51.592294 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591821 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 23:06:51.592294 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591823 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 23:06:51.592294 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591826 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 23:06:51.592294 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591829 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 23:06:51.592294 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591831 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 23:06:51.592294 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591833 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 23:06:51.592294 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591837 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 23:06:51.592294 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591840 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 23:06:51.592294 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591842 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 23:06:51.592294 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591845 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 23:06:51.592294 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591847 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 23:06:51.592964 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591863 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 23:06:51.592964 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591865 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 23:06:51.592964 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591868 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 23:06:51.592964 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591871 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 23:06:51.592964 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591874 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 23:06:51.592964 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591876 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 23:06:51.592964 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591879 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 23:06:51.592964 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591882 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 23:06:51.592964 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591885 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 23:06:51.592964 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591887 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 23:06:51.592964 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591890 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 23:06:51.592964 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591892 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 23:06:51.592964 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591896 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 23:06:51.592964 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591899 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 23:06:51.592964 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591901 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 23:06:51.592964 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591904 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 23:06:51.592964 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591906 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 23:06:51.592964 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591909 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 23:06:51.592964 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591912 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 23:06:51.592964 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591914 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 23:06:51.593580 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591917 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 23:06:51.593580 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591920 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 23:06:51.593580 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591922 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 23:06:51.593580 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591924 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 23:06:51.593580 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591927 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 23:06:51.593580 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591929 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 23:06:51.593580 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591932 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 23:06:51.593580 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591934 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 23:06:51.593580 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591937 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 23:06:51.593580 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591939 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 23:06:51.593580 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591942 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 23:06:51.593580 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591944 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 23:06:51.593580 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591947 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 23:06:51.593580 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591949 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 23:06:51.593580 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591952 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 23:06:51.593580 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591955 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 23:06:51.593580 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591958 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 23:06:51.593580 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591962 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 23:06:51.593580 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591966 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 23:06:51.594082 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591969 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 23:06:51.594082 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591972 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 23:06:51.594082 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591974 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 23:06:51.594082 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591977 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 23:06:51.594082 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591980 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 23:06:51.594082 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591982 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 23:06:51.594082 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591986 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 23:06:51.594082 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591989 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 23:06:51.594082 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591991 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 23:06:51.594082 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591994 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 23:06:51.594082 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591996 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 23:06:51.594082 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.591999 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 23:06:51.594082 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.592001 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 23:06:51.594082 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.592004 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 23:06:51.594082 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.592006 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 23:06:51.594082 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.592008 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 23:06:51.594082 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.592011 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 23:06:51.594082 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.592014 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 23:06:51.594560 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594435 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 23:06:51.594560 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594446 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 23:06:51.594560 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594450 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 23:06:51.594560 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594454 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 23:06:51.594560 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594457 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 23:06:51.594560 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594460 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 23:06:51.594560 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594462 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 23:06:51.594560 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594465 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 23:06:51.594560 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594468 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 23:06:51.594560 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594470 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 23:06:51.594560 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594473 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 23:06:51.594560 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594480 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 23:06:51.594560 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594483 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 23:06:51.594560 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594485 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 23:06:51.594560 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594488 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 23:06:51.594560 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594490 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 23:06:51.594560 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594493 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 23:06:51.594560 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594496 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 23:06:51.594560 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594499 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 23:06:51.594560 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594502 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 23:06:51.595097 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594504 2561 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 23:06:51.595097 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594508 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 23:06:51.595097 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594510 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 23:06:51.595097 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594513 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 23:06:51.595097 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594518 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 23:06:51.595097 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594520 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 23:06:51.595097 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594523 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 23:06:51.595097 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594526 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 23:06:51.595097 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594529 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 23:06:51.595097 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594531 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 23:06:51.595097 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594534 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 23:06:51.595097 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594537 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 23:06:51.595097 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594539 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 23:06:51.595097 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594542 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 23:06:51.595097 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594546 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 23:06:51.595097 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594549 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 23:06:51.595097 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594554 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 23:06:51.595097 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594558 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 23:06:51.595097 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594561 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 23:06:51.595097 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594564 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 23:06:51.595583 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594566 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 23:06:51.595583 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594569 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 23:06:51.595583 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594572 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 23:06:51.595583 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594575 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 23:06:51.595583 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594577 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 23:06:51.595583 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594580 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 23:06:51.595583 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594582 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 23:06:51.595583 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594585 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 23:06:51.595583 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594590 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 23:06:51.595583 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594593 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 23:06:51.595583 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594595 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 23:06:51.595583 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594598 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 23:06:51.595583 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594600 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 23:06:51.595583 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594603 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 23:06:51.595583 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594606 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 23:06:51.595583 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594609 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 23:06:51.595583 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594611 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 23:06:51.595583 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594614 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 23:06:51.595583 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594616 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 23:06:51.595583 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594619 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 23:06:51.596080 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594626 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 23:06:51.596080 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594630 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 23:06:51.596080 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594633 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 23:06:51.596080 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594635 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 23:06:51.596080 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594638 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 23:06:51.596080 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594643 2561 feature_gate.go:328] unrecognized feature gate: Example Sep 09 23:06:51.596080 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594646 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 23:06:51.596080 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594648 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 23:06:51.596080 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594651 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 23:06:51.596080 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594653 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 23:06:51.596080 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594656 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 23:06:51.596080 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594659 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 23:06:51.596080 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594661 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 23:06:51.596080 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594666 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 23:06:51.596080 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594669 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 23:06:51.596080 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594671 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 23:06:51.596080 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594674 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 23:06:51.596080 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594676 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 23:06:51.596080 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594679 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 23:06:51.596559 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594684 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 23:06:51.596559 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594688 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 23:06:51.596559 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594691 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 23:06:51.596559 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594693 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 23:06:51.596559 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594696 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 23:06:51.596559 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594699 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 23:06:51.596559 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594703 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 23:06:51.596559 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594707 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 23:06:51.596559 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594711 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 23:06:51.596559 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594714 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 23:06:51.596559 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594717 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 23:06:51.596559 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594720 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 23:06:51.596559 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594723 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 23:06:51.596559 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594725 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 23:06:51.596559 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594728 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 23:06:51.596559 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594731 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 23:06:51.596559 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594733 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 23:06:51.596559 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594736 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 23:06:51.596559 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594741 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 23:06:51.597041 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.594744 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 23:06:51.597041 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595514 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 23:06:51.597041 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595526 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 23:06:51.597041 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595530 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 23:06:51.597041 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595535 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 23:06:51.597041 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595538 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 23:06:51.597041 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595542 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 23:06:51.597041 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595545 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 23:06:51.597041 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595548 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 23:06:51.597041 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595551 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 23:06:51.597041 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595553 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 23:06:51.597041 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595556 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 23:06:51.597041 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595559 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 23:06:51.597041 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595562 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 23:06:51.597041 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595565 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 23:06:51.597041 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595568 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 23:06:51.597041 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595570 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 23:06:51.597041 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595573 2561 feature_gate.go:328] unrecognized feature gate: Example Sep 09 23:06:51.597041 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595575 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 23:06:51.597041 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595578 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 23:06:51.597527 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595580 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 23:06:51.597527 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595584 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 23:06:51.597527 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595587 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 23:06:51.597527 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595589 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 23:06:51.597527 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595592 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 23:06:51.597527 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595595 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 23:06:51.597527 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595597 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 23:06:51.597527 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595600 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 23:06:51.597527 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595602 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 23:06:51.597527 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595604 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 23:06:51.597527 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595607 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 23:06:51.597527 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595609 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 23:06:51.597527 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595612 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 23:06:51.597527 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595615 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 23:06:51.597527 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595618 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 23:06:51.597527 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595621 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 23:06:51.597527 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595623 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 23:06:51.597527 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595626 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 23:06:51.597527 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595628 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 23:06:51.597527 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595632 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 23:06:51.598037 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595635 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 23:06:51.598037 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595637 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 23:06:51.598037 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595639 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 23:06:51.598037 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595643 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 23:06:51.598037 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595645 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 23:06:51.598037 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595648 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 23:06:51.598037 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595650 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 23:06:51.598037 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595652 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 23:06:51.598037 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595655 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 23:06:51.598037 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595657 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 23:06:51.598037 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595660 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 23:06:51.598037 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595662 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 23:06:51.598037 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595665 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 23:06:51.598037 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595667 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 23:06:51.598037 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595670 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 23:06:51.598037 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595672 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 23:06:51.598037 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595675 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 23:06:51.598037 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595677 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 23:06:51.598037 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595680 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 23:06:51.598037 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595682 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 23:06:51.598506 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595685 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 23:06:51.598506 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595687 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 23:06:51.598506 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595690 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 23:06:51.598506 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595692 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 23:06:51.598506 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595695 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 23:06:51.598506 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595697 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 23:06:51.598506 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595701 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 23:06:51.598506 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595704 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 23:06:51.598506 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595706 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 23:06:51.598506 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595708 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 23:06:51.598506 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595711 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 23:06:51.598506 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595714 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 23:06:51.598506 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595716 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 23:06:51.598506 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595720 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 23:06:51.598506 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595724 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 23:06:51.598506 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595726 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 23:06:51.598506 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595729 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 23:06:51.598506 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595732 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 23:06:51.598506 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595735 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 23:06:51.598980 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595737 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 23:06:51.598980 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595740 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 23:06:51.598980 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595742 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 23:06:51.598980 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595745 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 23:06:51.598980 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595748 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 23:06:51.598980 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595751 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 23:06:51.598980 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595753 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 23:06:51.598980 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595756 2561 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 23:06:51.598980 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595758 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 23:06:51.598980 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595761 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 23:06:51.598980 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595763 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 23:06:51.598980 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595766 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 23:06:51.598980 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595769 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 23:06:51.598980 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595771 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 23:06:51.598980 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595774 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 23:06:51.598980 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595778 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 23:06:51.598980 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595782 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 23:06:51.598980 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595785 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 23:06:51.598980 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595788 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 23:06:51.599462 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595791 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 23:06:51.599462 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.595794 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 23:06:51.599462 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.595889 2561 flags.go:64] FLAG: --address="0.0.0.0" Sep 09 23:06:51.599462 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.595927 2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 09 23:06:51.599462 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.595934 2561 flags.go:64] FLAG: --anonymous-auth="true" Sep 09 23:06:51.599462 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.595938 2561 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 09 23:06:51.599462 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.595943 2561 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 09 23:06:51.599462 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.595947 2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 09 23:06:51.599462 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.595951 2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 09 23:06:51.599462 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.595956 2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 09 23:06:51.599462 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.595959 2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 09 23:06:51.599462 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.595962 2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 09 23:06:51.599462 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.595966 2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 09 23:06:51.599462 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.595969 2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 09 23:06:51.599462 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.595972 2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 09 23:06:51.599462 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.595975 2561 flags.go:64] FLAG: --cgroup-root="" Sep 09 23:06:51.599462 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.595978 2561 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 09 23:06:51.599462 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.595981 2561 flags.go:64] FLAG: --client-ca-file="" Sep 09 23:06:51.599462 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.595984 2561 flags.go:64] FLAG: --cloud-provider="external" Sep 09 23:06:51.599462 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.595987 2561 flags.go:64] FLAG: --cluster-dns="[]" Sep 09 23:06:51.599462 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.595991 2561 flags.go:64] FLAG: --cluster-domain="" Sep 09 23:06:51.599462 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.595994 2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 09 23:06:51.599462 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.595997 2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 09 23:06:51.600057 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596000 2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 09 23:06:51.600057 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596004 2561 flags.go:64] FLAG: --container-log-max-files="5" Sep 09 23:06:51.600057 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596007 2561 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 09 23:06:51.600057 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596010 2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 09 23:06:51.600057 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596014 2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 09 23:06:51.600057 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596017 2561 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 09 23:06:51.600057 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596020 2561 flags.go:64] FLAG: --contention-profiling="false" Sep 09 23:06:51.600057 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596022 2561 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 09 23:06:51.600057 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596025 2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 09 23:06:51.600057 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596029 2561 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 09 23:06:51.600057 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596033 2561 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 09 23:06:51.600057 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596037 2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 09 23:06:51.600057 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596040 2561 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 09 23:06:51.600057 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596043 2561 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 09 23:06:51.600057 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596046 2561 flags.go:64] FLAG: --enable-load-reader="false" Sep 09 23:06:51.600057 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596049 2561 flags.go:64] FLAG: --enable-server="true" Sep 09 23:06:51.600057 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596052 2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 09 23:06:51.600057 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596056 2561 flags.go:64] FLAG: --event-burst="100" Sep 09 23:06:51.600057 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596059 2561 flags.go:64] FLAG: --event-qps="50" Sep 09 23:06:51.600057 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596062 2561 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 09 23:06:51.600057 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596065 2561 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 09 23:06:51.600057 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596068 2561 flags.go:64] FLAG: --eviction-hard="" Sep 09 23:06:51.600057 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596071 2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 09 23:06:51.600057 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596074 2561 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 09 23:06:51.600641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596077 2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 09 23:06:51.600641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596080 2561 flags.go:64] FLAG: --eviction-soft="" Sep 09 23:06:51.600641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596083 2561 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 09 23:06:51.600641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596086 2561 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 09 23:06:51.600641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596089 2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 09 23:06:51.600641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596092 2561 flags.go:64] FLAG: --experimental-mounter-path="" Sep 09 23:06:51.600641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596095 2561 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 09 23:06:51.600641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596098 2561 flags.go:64] FLAG: --fail-swap-on="true" Sep 09 23:06:51.600641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596100 2561 flags.go:64] FLAG: --feature-gates="" Sep 09 23:06:51.600641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596104 2561 flags.go:64] FLAG: --file-check-frequency="20s" Sep 09 23:06:51.600641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596107 2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 09 23:06:51.600641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596111 2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 09 23:06:51.600641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596114 2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 09 23:06:51.600641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596117 2561 flags.go:64] FLAG: --healthz-port="10248" Sep 09 23:06:51.600641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596120 2561 flags.go:64] FLAG: --help="false" Sep 09 23:06:51.600641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596122 2561 flags.go:64] FLAG: --hostname-override="ip-10-0-143-192.ec2.internal" Sep 09 23:06:51.600641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596125 2561 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 09 23:06:51.600641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596128 2561 flags.go:64] FLAG: --http-check-frequency="20s" Sep 09 23:06:51.600641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596131 2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 09 23:06:51.600641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596134 2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 09 23:06:51.600641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596138 2561 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 09 23:06:51.600641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596141 2561 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 09 23:06:51.600641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596143 2561 flags.go:64] FLAG: --image-service-endpoint="" Sep 09 23:06:51.601255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596146 2561 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 09 23:06:51.601255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596149 2561 flags.go:64] FLAG: --kube-api-burst="100" Sep 09 23:06:51.601255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596152 2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 09 23:06:51.601255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596155 2561 flags.go:64] FLAG: --kube-api-qps="50" Sep 09 23:06:51.601255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596158 2561 flags.go:64] FLAG: --kube-reserved="" Sep 09 23:06:51.601255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596160 2561 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 09 23:06:51.601255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596163 2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 09 23:06:51.601255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596166 2561 flags.go:64] FLAG: --kubelet-cgroups="" Sep 09 23:06:51.601255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596169 2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 09 23:06:51.601255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596171 2561 flags.go:64] FLAG: --lock-file="" Sep 09 23:06:51.601255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596174 2561 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 09 23:06:51.601255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596177 2561 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 09 23:06:51.601255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596180 2561 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 09 23:06:51.601255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596185 2561 flags.go:64] FLAG: --log-json-split-stream="false" Sep 09 23:06:51.601255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596188 2561 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 09 23:06:51.601255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596191 2561 flags.go:64] FLAG: --log-text-split-stream="false" Sep 09 23:06:51.601255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596194 2561 flags.go:64] FLAG: --logging-format="text" Sep 09 23:06:51.601255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596197 2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 09 23:06:51.601255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596200 2561 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 09 23:06:51.601255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596203 2561 flags.go:64] FLAG: --manifest-url="" Sep 09 23:06:51.601255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596206 2561 flags.go:64] FLAG: --manifest-url-header="" Sep 09 23:06:51.601255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596210 2561 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 09 23:06:51.601255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596213 2561 flags.go:64] FLAG: --max-open-files="1000000" Sep 09 23:06:51.601255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596217 2561 flags.go:64] FLAG: --max-pods="110" Sep 09 23:06:51.601255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596220 2561 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 09 23:06:51.601970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596222 2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 09 23:06:51.601970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596225 2561 flags.go:64] FLAG: --memory-manager-policy="None" Sep 09 23:06:51.601970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596228 2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 09 23:06:51.601970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596231 2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 09 23:06:51.601970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596234 2561 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 09 23:06:51.601970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596237 2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 09 23:06:51.601970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596244 2561 flags.go:64] FLAG: --node-status-max-images="50" Sep 09 23:06:51.601970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596248 2561 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 09 23:06:51.601970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596251 2561 flags.go:64] FLAG: --oom-score-adj="-999" Sep 09 23:06:51.601970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596254 2561 flags.go:64] FLAG: --pod-cidr="" Sep 09 23:06:51.601970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596256 2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 09 23:06:51.601970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596263 2561 flags.go:64] FLAG: --pod-manifest-path="" Sep 09 23:06:51.601970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596266 2561 flags.go:64] FLAG: --pod-max-pids="-1" Sep 09 23:06:51.601970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596269 2561 flags.go:64] FLAG: --pods-per-core="0" Sep 09 23:06:51.601970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596271 2561 flags.go:64] FLAG: --port="10250" Sep 09 23:06:51.601970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596274 2561 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 09 23:06:51.601970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596277 2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d02de2e90f162118" Sep 09 23:06:51.601970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596280 2561 flags.go:64] FLAG: --qos-reserved="" Sep 09 23:06:51.601970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596283 2561 flags.go:64] FLAG: --read-only-port="10255" Sep 09 23:06:51.601970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596286 2561 flags.go:64] FLAG: --register-node="true" Sep 09 23:06:51.601970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596289 2561 flags.go:64] FLAG: --register-with-taints="" Sep 09 23:06:51.601970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596297 2561 flags.go:64] FLAG: --registry-burst="10" Sep 09 23:06:51.601970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596300 2561 flags.go:64] FLAG: --registry-qps="5" Sep 09 23:06:51.601970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596303 2561 flags.go:64] FLAG: --reserved-cpus="" Sep 09 23:06:51.602579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596306 2561 flags.go:64] FLAG: --reserved-memory="" Sep 09 23:06:51.602579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596310 2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 09 23:06:51.602579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596313 2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 09 23:06:51.602579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596315 2561 flags.go:64] FLAG: --rotate-certificates="false" Sep 09 23:06:51.602579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596319 2561 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 09 23:06:51.602579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596321 2561 flags.go:64] FLAG: --runonce="false" Sep 09 23:06:51.602579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596324 2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 09 23:06:51.602579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596327 2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 09 23:06:51.602579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596330 2561 flags.go:64] FLAG: --seccomp-default="false" Sep 09 23:06:51.602579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596333 2561 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 09 23:06:51.602579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596336 2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 09 23:06:51.602579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596339 2561 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 09 23:06:51.602579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596341 2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 09 23:06:51.602579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596344 2561 flags.go:64] FLAG: --storage-driver-password="root" Sep 09 23:06:51.602579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596347 2561 flags.go:64] FLAG: --storage-driver-secure="false" Sep 09 23:06:51.602579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596350 2561 flags.go:64] FLAG: --storage-driver-table="stats" Sep 09 23:06:51.602579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596353 2561 flags.go:64] FLAG: --storage-driver-user="root" Sep 09 23:06:51.602579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596355 2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 09 23:06:51.602579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596358 2561 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 09 23:06:51.602579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596361 2561 flags.go:64] FLAG: --system-cgroups="" Sep 09 23:06:51.602579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596363 2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 09 23:06:51.602579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596369 2561 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 09 23:06:51.602579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596371 2561 flags.go:64] FLAG: --tls-cert-file="" Sep 09 23:06:51.602579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596374 2561 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 09 23:06:51.602579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596377 2561 flags.go:64] FLAG: --tls-min-version="" Sep 09 23:06:51.603265 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596380 2561 flags.go:64] FLAG: --tls-private-key-file="" Sep 09 23:06:51.603265 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596383 2561 flags.go:64] FLAG: --topology-manager-policy="none" Sep 09 23:06:51.603265 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596385 2561 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 09 23:06:51.603265 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596388 2561 flags.go:64] FLAG: --topology-manager-scope="container" Sep 09 23:06:51.603265 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596391 2561 flags.go:64] FLAG: --v="2" Sep 09 23:06:51.603265 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596395 2561 flags.go:64] FLAG: --version="false" Sep 09 23:06:51.603265 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596399 2561 flags.go:64] FLAG: --vmodule="" Sep 09 23:06:51.603265 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596402 2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 09 23:06:51.603265 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.596408 2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 09 23:06:51.603265 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596523 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 23:06:51.603265 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596526 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 23:06:51.603265 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596531 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 23:06:51.603265 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596534 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 23:06:51.603265 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596538 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 23:06:51.603265 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596540 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 23:06:51.603265 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596543 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 23:06:51.603265 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596546 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 23:06:51.603265 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596549 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 23:06:51.603265 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596552 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 23:06:51.603265 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596556 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 23:06:51.603265 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596558 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 23:06:51.603801 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596561 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 23:06:51.603801 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596564 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 23:06:51.603801 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596567 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 23:06:51.603801 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596569 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 23:06:51.603801 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596572 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 23:06:51.603801 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596574 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 23:06:51.603801 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596577 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 23:06:51.603801 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596579 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 23:06:51.603801 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596583 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 23:06:51.603801 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596585 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 23:06:51.603801 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596587 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 23:06:51.603801 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596590 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 23:06:51.603801 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596592 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 23:06:51.603801 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596595 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 23:06:51.603801 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596597 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 23:06:51.603801 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596600 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 23:06:51.603801 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596602 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 23:06:51.603801 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596605 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 23:06:51.603801 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596608 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 23:06:51.603801 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596611 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 23:06:51.604314 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596614 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 23:06:51.604314 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596616 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 23:06:51.604314 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596619 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 23:06:51.604314 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596622 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 23:06:51.604314 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596624 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 23:06:51.604314 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596627 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 23:06:51.604314 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596629 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 23:06:51.604314 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596632 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 23:06:51.604314 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596634 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 23:06:51.604314 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596636 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 23:06:51.604314 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596639 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 23:06:51.604314 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596642 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 23:06:51.604314 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596644 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 23:06:51.604314 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596647 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 23:06:51.604314 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596649 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 23:06:51.604314 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596652 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 23:06:51.604314 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596654 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 23:06:51.604314 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596657 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 23:06:51.604314 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596659 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 23:06:51.604314 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596661 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 23:06:51.604813 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596664 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 23:06:51.604813 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596666 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 23:06:51.604813 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596669 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 23:06:51.604813 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596671 2561 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 23:06:51.604813 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596673 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 23:06:51.604813 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596676 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 23:06:51.604813 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596678 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 23:06:51.604813 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596680 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 23:06:51.604813 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596683 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 23:06:51.604813 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596685 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 23:06:51.604813 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596688 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 23:06:51.604813 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596692 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 23:06:51.604813 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596695 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 23:06:51.604813 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596697 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 23:06:51.604813 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596702 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 23:06:51.604813 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596705 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 23:06:51.604813 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596707 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 23:06:51.604813 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596710 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 23:06:51.604813 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596712 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 23:06:51.604813 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596715 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 23:06:51.605346 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596718 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 23:06:51.605346 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596720 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 23:06:51.605346 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596723 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 23:06:51.605346 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596725 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 23:06:51.605346 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596728 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 23:06:51.605346 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596730 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 23:06:51.605346 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596733 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 23:06:51.605346 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596735 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 23:06:51.605346 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596737 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 23:06:51.605346 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596739 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 23:06:51.605346 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596742 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 23:06:51.605346 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596745 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 23:06:51.605346 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596748 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 23:06:51.605346 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596751 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 23:06:51.605346 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596753 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 23:06:51.605346 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596756 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 23:06:51.605346 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596758 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 23:06:51.605346 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596760 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 23:06:51.605346 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596763 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 23:06:51.605830 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596765 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 23:06:51.605830 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596768 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 23:06:51.605830 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596770 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 23:06:51.605830 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596772 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 23:06:51.605830 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596776 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 23:06:51.605830 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596780 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 23:06:51.605830 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596782 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 23:06:51.605830 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.596785 2561 feature_gate.go:328] unrecognized feature gate: Example Sep 09 23:06:51.605830 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.597264 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 23:06:51.605830 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.605137 2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 09 23:06:51.605830 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.605161 2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 09 23:06:51.606130 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.605974 2561 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 09 23:06:51.606130 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.605983 2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 09 23:06:51.606130 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606055 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 23:06:51.606130 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606062 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 23:06:51.606130 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606067 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 23:06:51.606130 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606070 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 23:06:51.606130 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606073 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 23:06:51.606130 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606077 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 23:06:51.606130 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606079 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 23:06:51.606130 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606082 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 23:06:51.606130 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606085 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 23:06:51.606130 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606087 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 23:06:51.606130 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606089 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 23:06:51.606130 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606092 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 23:06:51.606130 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606095 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 23:06:51.606130 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606097 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 23:06:51.606130 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606100 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 23:06:51.606130 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606102 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 23:06:51.606130 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606105 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 23:06:51.606631 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606107 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 23:06:51.606631 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606110 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 23:06:51.606631 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606112 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 23:06:51.606631 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606115 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 23:06:51.606631 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606117 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 23:06:51.606631 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606120 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 23:06:51.606631 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606123 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 23:06:51.606631 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606125 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 23:06:51.606631 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606128 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 23:06:51.606631 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606130 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 23:06:51.606631 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606133 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 23:06:51.606631 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606136 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 23:06:51.606631 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606140 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 23:06:51.606631 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606143 2561 feature_gate.go:328] unrecognized feature gate: Example Sep 09 23:06:51.606631 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606146 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 23:06:51.606631 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606148 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 23:06:51.606631 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606151 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 23:06:51.606631 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606153 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 23:06:51.606631 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606155 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 23:06:51.606631 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606157 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 23:06:51.607229 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606160 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 23:06:51.607229 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606162 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 23:06:51.607229 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606165 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 23:06:51.607229 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606167 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 23:06:51.607229 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606170 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 23:06:51.607229 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606172 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 23:06:51.607229 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606174 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 23:06:51.607229 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606177 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 23:06:51.607229 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606179 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 23:06:51.607229 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606181 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 23:06:51.607229 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606184 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 23:06:51.607229 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606187 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 23:06:51.607229 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606190 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 23:06:51.607229 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606193 2561 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 23:06:51.607229 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606196 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 23:06:51.607229 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606198 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 23:06:51.607229 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606202 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 23:06:51.607229 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606207 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 23:06:51.607229 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606210 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 23:06:51.607229 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606213 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 23:06:51.607715 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606215 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 23:06:51.607715 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606217 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 23:06:51.607715 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606220 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 23:06:51.607715 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606222 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 23:06:51.607715 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606225 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 23:06:51.607715 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606228 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 23:06:51.607715 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606230 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 23:06:51.607715 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606233 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 23:06:51.607715 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606235 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 23:06:51.607715 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606238 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 23:06:51.607715 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606240 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 23:06:51.607715 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606243 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 23:06:51.607715 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606245 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 23:06:51.607715 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606247 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 23:06:51.607715 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606250 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 23:06:51.607715 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606252 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 23:06:51.607715 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606255 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 23:06:51.607715 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606257 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 23:06:51.607715 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606260 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 23:06:51.607715 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606262 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 23:06:51.608307 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606265 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 23:06:51.608307 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606269 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 23:06:51.608307 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606272 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 23:06:51.608307 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606275 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 23:06:51.608307 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606277 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 23:06:51.608307 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606280 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 23:06:51.608307 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606283 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 23:06:51.608307 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606285 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 23:06:51.608307 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606288 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 23:06:51.608307 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606290 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 23:06:51.608307 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606293 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 23:06:51.608307 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606295 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 23:06:51.608307 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606298 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 23:06:51.608307 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606301 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 23:06:51.608307 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606303 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 23:06:51.608307 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606306 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 23:06:51.608307 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606308 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 23:06:51.608307 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606311 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 23:06:51.608307 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606314 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 23:06:51.608765 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606316 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 23:06:51.608765 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606319 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 23:06:51.608765 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606321 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 23:06:51.608765 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.606326 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 23:06:51.608765 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606453 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 23:06:51.608765 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606459 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 23:06:51.608765 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606464 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 23:06:51.608765 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606467 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 23:06:51.608765 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606470 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 23:06:51.608765 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606472 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 23:06:51.608765 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606475 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 23:06:51.608765 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606477 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 23:06:51.608765 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606480 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 23:06:51.608765 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606483 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 23:06:51.609131 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606486 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 23:06:51.609131 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606488 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 23:06:51.609131 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606490 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 23:06:51.609131 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606495 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 23:06:51.609131 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606497 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 23:06:51.609131 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606500 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 23:06:51.609131 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606503 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 23:06:51.609131 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606505 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 23:06:51.609131 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606509 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 23:06:51.609131 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606512 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 23:06:51.609131 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606515 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 23:06:51.609131 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606518 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 23:06:51.609131 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606520 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 23:06:51.609131 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606523 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 23:06:51.609131 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606526 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 23:06:51.609131 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606528 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 23:06:51.609131 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606531 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 23:06:51.609131 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606533 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 23:06:51.609131 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606540 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 23:06:51.609587 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606543 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 23:06:51.609587 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606545 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 23:06:51.609587 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606548 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 23:06:51.609587 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606550 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 23:06:51.609587 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606553 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 23:06:51.609587 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606555 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 23:06:51.609587 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606558 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 23:06:51.609587 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606560 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 23:06:51.609587 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606562 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 23:06:51.609587 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606565 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 23:06:51.609587 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606567 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 23:06:51.609587 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606570 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 23:06:51.609587 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606572 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 23:06:51.609587 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606575 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 23:06:51.609587 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606577 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 23:06:51.609587 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606580 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 23:06:51.609587 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606583 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 23:06:51.609587 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606586 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 23:06:51.609587 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606589 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 23:06:51.609587 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606592 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 23:06:51.610098 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606594 2561 feature_gate.go:328] unrecognized feature gate: Example Sep 09 23:06:51.610098 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606597 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 23:06:51.610098 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606599 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 23:06:51.610098 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606602 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 23:06:51.610098 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606604 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 23:06:51.610098 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606607 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 23:06:51.610098 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606610 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 23:06:51.610098 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606613 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 23:06:51.610098 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606615 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 23:06:51.610098 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606618 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 23:06:51.610098 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606620 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 23:06:51.610098 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606623 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 23:06:51.610098 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606626 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 23:06:51.610098 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606629 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 23:06:51.610098 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606632 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 23:06:51.610098 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606634 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 23:06:51.610098 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606637 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 23:06:51.610098 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606639 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 23:06:51.610098 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606642 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 23:06:51.610098 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606644 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 23:06:51.610586 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606648 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 23:06:51.610586 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606650 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 23:06:51.610586 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606653 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 23:06:51.610586 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606655 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 23:06:51.610586 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606658 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 23:06:51.610586 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606660 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 23:06:51.610586 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606663 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 23:06:51.610586 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606665 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 23:06:51.610586 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606668 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 23:06:51.610586 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606671 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 23:06:51.610586 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606675 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 23:06:51.610586 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606678 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 23:06:51.610586 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606681 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 23:06:51.610586 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606684 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 23:06:51.610586 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606687 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 23:06:51.610586 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606690 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 23:06:51.610586 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606692 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 23:06:51.610586 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606695 2561 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 23:06:51.610586 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606697 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 23:06:51.611068 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606700 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 23:06:51.611068 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606702 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 23:06:51.611068 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606704 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 23:06:51.611068 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606707 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 23:06:51.611068 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606709 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 23:06:51.611068 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606711 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 23:06:51.611068 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606714 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 23:06:51.611068 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606717 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 23:06:51.611068 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606719 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 23:06:51.611068 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606721 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 23:06:51.611068 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:51.606724 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 23:06:51.611068 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.606729 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 23:06:51.611068 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.606886 2561 server.go:962] "Client rotation is on, will bootstrap in background" Sep 09 23:06:51.611068 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.608907 2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 09 23:06:51.611068 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.609771 2561 server.go:1019] "Starting client certificate rotation" Sep 09 23:06:51.611423 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.609886 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 23:06:51.611423 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.609926 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 23:06:51.612338 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.612322 2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 23:06:51.613494 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.613467 2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 23:06:51.624014 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.623993 2561 log.go:25] "Validated CRI v1 runtime API" Sep 09 23:06:51.626845 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.626828 2561 log.go:25] "Validated CRI v1 image API" Sep 09 23:06:51.627718 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.627703 2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 09 23:06:51.631982 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.631957 2561 fs.go:135] Filesystem UUIDs: map[6cce1dc2-cced-4875-a2a4-d2da780994f2:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ea922eaf-5763-4718-9d41-ad6150fb12c8:/dev/nvme0n1p4] Sep 09 23:06:51.631982 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.631979 2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 09 23:06:51.635394 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.635374 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 23:06:51.638005 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.637885 2561 manager.go:216] Machine: {Timestamp:2026-09-09 23:06:51.636529009 +0000 UTC m=+0.322312423 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099313 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec297d26c71417f40f6ee7ca7faf3cfb SystemUUID:ec297d26-c714-17f4-0f6e-e7ca7faf3cfb BootID:a6d49140-f985-48e1-866b-4a8a572964f6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:de:1a:59:b0:f7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:de:1a:59:b0:f7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:fa:3b:48:5f:da Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 09 23:06:51.638005 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.637999 2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 09 23:06:51.638122 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.638091 2561 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 09 23:06:51.641047 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.641019 2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 09 23:06:51.641192 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.641050 2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-192.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 09 23:06:51.642169 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.642157 2561 topology_manager.go:138] "Creating topology manager with none policy" Sep 09 23:06:51.642202 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.642172 2561 container_manager_linux.go:309] "Creating device plugin manager" Sep 09 23:06:51.642202 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.642180 2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 23:06:51.642879 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.642866 2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 23:06:51.644057 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.644046 2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 09 23:06:51.648183 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.648166 2561 state_mem.go:36] "Initialized new in-memory state store" Sep 09 23:06:51.648320 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.648310 2561 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 09 23:06:51.648353 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.648345 2561 kubelet.go:486] "Attempting to sync node with API server" Sep 09 23:06:51.648384 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.648368 2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 09 23:06:51.648416 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.648388 2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 09 23:06:51.648416 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.648401 2561 kubelet.go:398] "Adding apiserver pod source" Sep 09 23:06:51.648416 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.648413 2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 09 23:06:51.648836 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.648821 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 23:06:51.648923 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.648845 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 23:06:51.651168 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.651147 2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 09 23:06:51.653199 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.653184 2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 09 23:06:51.653242 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.653232 2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 09 23:06:51.653431 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.653421 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 09 23:06:51.653463 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.653437 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 09 23:06:51.653463 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.653443 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 09 23:06:51.653463 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.653448 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 09 23:06:51.653463 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.653456 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 09 23:06:51.653463 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.653461 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 09 23:06:51.653965 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.653955 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 09 23:06:51.653965 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.653965 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 09 23:06:51.654030 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.653973 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 09 23:06:51.654030 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.653978 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 09 23:06:51.654030 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.653998 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 09 23:06:51.654030 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.654009 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 09 23:06:51.655481 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.655466 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 09 23:06:51.655516 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.655483 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 09 23:06:51.658742 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.658711 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wdppq" Sep 09 23:06:51.659628 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.659616 2561 server.go:1268] "Started kubelet" Sep 09 23:06:51.659747 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.659724 2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 09 23:06:51.659801 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.659779 2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 09 23:06:51.659846 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.659825 2561 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 09 23:06:51.660837 ip-10-0-143-192 systemd[1]: Started Kubernetes Kubelet. Sep 09 23:06:51.660988 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.660923 2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-192.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 09 23:06:51.661326 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:51.661296 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 09 23:06:51.661410 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:51.661297 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-192.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 09 23:06:51.661469 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.661424 2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 09 23:06:51.662617 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.662420 2561 server.go:310] "Adding debug handlers to kubelet server" Sep 09 23:06:51.664333 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:51.664310 2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 09 23:06:51.666233 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.666047 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wdppq" Sep 09 23:06:51.666559 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.666543 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 09 23:06:51.667124 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.667110 2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 09 23:06:51.667239 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:51.667220 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Sep 09 23:06:51.667678 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.667661 2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 09 23:06:51.667678 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.667667 2561 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 09 23:06:51.667815 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.667702 2561 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 09 23:06:51.667815 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:51.666829 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-192.ec2.internal.18d3c9aa9fd8256b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-192.ec2.internal,UID:ip-10-0-143-192.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-192.ec2.internal,},FirstTimestamp:2026-09-09 23:06:51.659584875 +0000 UTC m=+0.345368268,LastTimestamp:2026-09-09 23:06:51.659584875 +0000 UTC m=+0.345368268,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-192.ec2.internal,}" Sep 09 23:06:51.667950 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.667820 2561 reconstruct.go:97] "Volume reconstruction finished" Sep 09 23:06:51.667950 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.667829 2561 reconciler.go:29] "Reconciler: start to sync state" Sep 09 23:06:51.668248 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.668236 2561 factory.go:153] Registering CRI-O factory Sep 09 23:06:51.668292 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.668253 2561 factory.go:223] Registration of the crio container factory successfully Sep 09 23:06:51.668340 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.668295 2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 09 23:06:51.668340 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.668303 2561 factory.go:55] Registering systemd factory Sep 09 23:06:51.668340 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.668308 2561 factory.go:223] Registration of the systemd container factory successfully Sep 09 23:06:51.668340 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.668320 2561 factory.go:103] Registering Raw factory Sep 09 23:06:51.668340 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.668328 2561 manager.go:1196] Started watching for new ooms in manager Sep 09 23:06:51.668768 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.668672 2561 manager.go:349] Starting recovery of all containers Sep 09 23:06:51.675584 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.675560 2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 23:06:51.679257 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.679188 2561 manager.go:354] Recovery completed Sep 09 23:06:51.679763 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:51.679741 2561 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-192.ec2.internal\" not found" node="ip-10-0-143-192.ec2.internal" Sep 09 23:06:51.683991 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.683976 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:06:51.688397 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.688377 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:06:51.688495 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.688414 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:06:51.688495 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.688429 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:06:51.689167 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.689152 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:06:51.689246 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.689151 2561 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 09 23:06:51.689295 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.689233 2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 09 23:06:51.689295 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.689268 2561 state_mem.go:36] "Initialized new in-memory state store" Sep 09 23:06:51.690048 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.690033 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:06:51.690125 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.690066 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:06:51.690125 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.690080 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:06:51.691350 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.691338 2561 policy_none.go:49] "None policy: Start" Sep 09 23:06:51.691397 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.691355 2561 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 09 23:06:51.691397 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.691366 2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 09 23:06:51.693332 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.693321 2561 policy_none.go:47] "Start" Sep 09 23:06:51.745975 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.728525 2561 manager.go:337] "Starting Device Plugin manager" Sep 09 23:06:51.745975 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:51.728565 2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 09 23:06:51.745975 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.728577 2561 server.go:87] "Starting device plugin registration server" Sep 09 23:06:51.745975 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.728964 2561 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 09 23:06:51.745975 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.728977 2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 09 23:06:51.745975 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.729078 2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 09 23:06:51.745975 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.729183 2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 09 23:06:51.745975 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.729193 2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 09 23:06:51.745975 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:51.729729 2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 09 23:06:51.745975 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:51.729793 2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-192.ec2.internal\" not found" Sep 09 23:06:51.801741 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.801703 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 09 23:06:51.803167 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.803137 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 09 23:06:51.803281 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.803195 2561 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 09 23:06:51.803281 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.803219 2561 kubelet.go:2448] "Starting kubelet main sync loop" Sep 09 23:06:51.803281 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:51.803268 2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 09 23:06:51.806936 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.806916 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 23:06:51.829804 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.829775 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:06:51.830845 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.830827 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:06:51.830941 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.830883 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:06:51.830941 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.830895 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:06:51.830941 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.830921 2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-192.ec2.internal" Sep 09 23:06:51.837003 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.836927 2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-192.ec2.internal" Sep 09 23:06:51.837003 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:51.836954 2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-192.ec2.internal\": node \"ip-10-0-143-192.ec2.internal\" not found" Sep 09 23:06:51.850206 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:51.850175 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Sep 09 23:06:51.903673 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.903622 2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-192.ec2.internal"] Sep 09 23:06:51.903765 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.903730 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:06:51.905409 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.905391 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:06:51.905516 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.905421 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:06:51.905516 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.905432 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:06:51.907776 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.907763 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:06:51.907936 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.907921 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" Sep 09 23:06:51.907977 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.907956 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:06:51.908599 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.908580 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:06:51.908650 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.908610 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:06:51.908650 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.908621 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:06:51.908710 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.908591 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:06:51.908710 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.908680 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:06:51.908710 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.908691 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:06:51.910757 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.910743 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-192.ec2.internal" Sep 09 23:06:51.910835 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.910770 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:06:51.911539 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.911519 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:06:51.911626 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.911558 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:06:51.911626 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.911574 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:06:51.927533 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:51.927506 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-192.ec2.internal\" not found" node="ip-10-0-143-192.ec2.internal" Sep 09 23:06:51.930937 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:51.930918 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-192.ec2.internal\" not found" node="ip-10-0-143-192.ec2.internal" Sep 09 23:06:51.950423 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:51.950394 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Sep 09 23:06:51.970382 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.970345 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/47e768c654fbb38de18e4644c0ab2928-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal\" (UID: \"47e768c654fbb38de18e4644c0ab2928\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" Sep 09 23:06:51.970382 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.970384 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47e768c654fbb38de18e4644c0ab2928-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal\" (UID: \"47e768c654fbb38de18e4644c0ab2928\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" Sep 09 23:06:51.970568 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:51.970404 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef36ec20e9fe178e1d7d8c3ae65f1ea2-config\") pod \"kube-apiserver-proxy-ip-10-0-143-192.ec2.internal\" (UID: \"ef36ec20e9fe178e1d7d8c3ae65f1ea2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-192.ec2.internal" Sep 09 23:06:52.050786 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:52.050738 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Sep 09 23:06:52.071155 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:52.071122 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47e768c654fbb38de18e4644c0ab2928-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal\" (UID: \"47e768c654fbb38de18e4644c0ab2928\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" Sep 09 23:06:52.071255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:52.071163 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef36ec20e9fe178e1d7d8c3ae65f1ea2-config\") pod \"kube-apiserver-proxy-ip-10-0-143-192.ec2.internal\" (UID: \"ef36ec20e9fe178e1d7d8c3ae65f1ea2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-192.ec2.internal" Sep 09 23:06:52.071255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:52.071186 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/47e768c654fbb38de18e4644c0ab2928-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal\" (UID: \"47e768c654fbb38de18e4644c0ab2928\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" Sep 09 23:06:52.071356 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:52.071254 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef36ec20e9fe178e1d7d8c3ae65f1ea2-config\") pod \"kube-apiserver-proxy-ip-10-0-143-192.ec2.internal\" (UID: \"ef36ec20e9fe178e1d7d8c3ae65f1ea2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-192.ec2.internal" Sep 09 23:06:52.071356 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:52.071296 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/47e768c654fbb38de18e4644c0ab2928-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal\" (UID: \"47e768c654fbb38de18e4644c0ab2928\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" Sep 09 23:06:52.071356 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:52.071255 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47e768c654fbb38de18e4644c0ab2928-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal\" (UID: \"47e768c654fbb38de18e4644c0ab2928\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" Sep 09 23:06:52.151641 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:52.151564 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Sep 09 23:06:52.230093 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:52.230050 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" Sep 09 23:06:52.233225 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:52.233195 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-192.ec2.internal" Sep 09 23:06:52.252061 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:52.252023 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Sep 09 23:06:52.352593 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:52.352549 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Sep 09 23:06:52.453179 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:52.453089 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Sep 09 23:06:52.553677 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:52.553645 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Sep 09 23:06:52.610247 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:52.610155 2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 09 23:06:52.610731 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:52.610412 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 23:06:52.610731 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:52.610425 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 23:06:52.654703 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:52.654662 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Sep 09 23:06:52.669699 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:52.669652 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 23:01:52 +0000 UTC" deadline="2028-04-13 04:05:13.459203182 +0000 UTC" Sep 09 23:06:52.669699 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:52.669688 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13948h58m20.789517973s" Sep 09 23:06:52.755006 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:52.754972 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Sep 09 23:06:52.783339 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:52.783310 2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 23:06:52.844470 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:52.844435 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef36ec20e9fe178e1d7d8c3ae65f1ea2.slice/crio-65bffe6fbf2a292dd39c5eaa4e442801aa82dc51ba9220c5e2ff7119c2517436 WatchSource:0}: Error finding container 65bffe6fbf2a292dd39c5eaa4e442801aa82dc51ba9220c5e2ff7119c2517436: Status 404 returned error can't find the container with id 65bffe6fbf2a292dd39c5eaa4e442801aa82dc51ba9220c5e2ff7119c2517436 Sep 09 23:06:52.850923 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:52.850900 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 23:06:52.855764 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:52.855741 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Sep 09 23:06:52.857349 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:52.857319 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e768c654fbb38de18e4644c0ab2928.slice/crio-906cb28ce07c5bfbe0dd4f1948966c1e38b8d705502f188499421494106039c2 WatchSource:0}: Error finding container 906cb28ce07c5bfbe0dd4f1948966c1e38b8d705502f188499421494106039c2: Status 404 returned error can't find the container with id 906cb28ce07c5bfbe0dd4f1948966c1e38b8d705502f188499421494106039c2 Sep 09 23:06:52.937665 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:52.937624 2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 23:06:52.967970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:52.967930 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-192.ec2.internal" Sep 09 23:06:52.978364 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:52.978289 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 23:06:52.979286 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:52.979272 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" Sep 09 23:06:52.988443 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:52.988424 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 23:06:53.650096 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.650058 2561 apiserver.go:52] "Watching apiserver" Sep 09 23:06:53.660577 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.660536 2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 09 23:06:53.661498 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.661063 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-gw2dp","kube-system/kube-apiserver-proxy-ip-10-0-143-192.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k","openshift-cluster-node-tuning-operator/tuned-5qpkc","openshift-dns/node-resolver-jqxbd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal","openshift-multus/multus-t69sp","openshift-network-diagnostics/network-check-target-v7c8p","kube-system/konnectivity-agent-z9xqn","openshift-image-registry/node-ca-d76sb","openshift-multus/multus-additional-cni-plugins-zncr7","openshift-multus/network-metrics-daemon-j89rl","openshift-network-operator/iptables-alerter-25rz7","openshift-ovn-kubernetes/ovnkube-node-pbrgn"] Sep 09 23:06:53.663863 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.663838 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.666355 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.666330 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-25rz7" Sep 09 23:06:53.666465 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.666410 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 09 23:06:53.672450 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.672421 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jd6js\"" Sep 09 23:06:53.672570 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.672468 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" Sep 09 23:06:53.675239 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.672822 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 09 23:06:53.675239 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.672951 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 09 23:06:53.675239 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.673104 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.675239 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.673183 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 09 23:06:53.675239 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.673380 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 09 23:06:53.675239 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.674307 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jqxbd" Sep 09 23:06:53.675239 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.674471 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 09 23:06:53.675239 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.674517 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 09 23:06:53.675239 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.674579 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dsb9k\"" Sep 09 23:06:53.675239 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.675041 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 09 23:06:53.675808 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.675532 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 09 23:06:53.675954 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.675915 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kthg9\"" Sep 09 23:06:53.676091 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.676057 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 09 23:06:53.676759 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.676367 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 09 23:06:53.676759 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.676554 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4nxc9\"" Sep 09 23:06:53.676759 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.676599 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 09 23:06:53.677564 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.677544 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 09 23:06:53.677749 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.677733 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 09 23:06:53.677800 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.677779 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xm2gn\"" Sep 09 23:06:53.678747 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.678023 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 09 23:06:53.678747 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.678318 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.679707 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.679685 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-host-run-multus-certs\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.679811 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.679721 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3881b43c-4c39-4800-a4fd-a060fb8cf5e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-pft9k\" (UID: \"3881b43c-4c39-4800-a4fd-a060fb8cf5e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" Sep 09 23:06:53.679811 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.679747 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3881b43c-4c39-4800-a4fd-a060fb8cf5e8-sys-fs\") pod \"aws-ebs-csi-driver-node-pft9k\" (UID: \"3881b43c-4c39-4800-a4fd-a060fb8cf5e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" Sep 09 23:06:53.679811 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.679772 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-multus-cni-dir\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.679811 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.679798 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-etc-modprobe-d\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.680054 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.679821 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-etc-systemd\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.680054 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.679846 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxc98\" (UniqueName: \"kubernetes.io/projected/08bcba00-9881-442c-aa02-b875c99817c2-kube-api-access-lxc98\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.680054 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.679872 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a843f1d8-0cd1-4396-8fc1-5a22cd20f6e8-tmp-dir\") pod \"node-resolver-jqxbd\" (UID: \"a843f1d8-0cd1-4396-8fc1-5a22cd20f6e8\") " pod="openshift-dns/node-resolver-jqxbd" Sep 09 23:06:53.680054 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.679904 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-cni-binary-copy\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.680255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680145 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-host-var-lib-kubelet\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.680255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680188 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-multus-daemon-config\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.680255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680218 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5560171c-fecb-4328-a381-d6ecb4a7bcaf-host-slash\") pod \"iptables-alerter-25rz7\" (UID: \"5560171c-fecb-4328-a381-d6ecb4a7bcaf\") " pod="openshift-network-operator/iptables-alerter-25rz7" Sep 09 23:06:53.680255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680244 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-multus-socket-dir-parent\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.680458 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680269 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-host-run-netns\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.680458 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680292 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-hostroot\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.680458 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680316 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5560171c-fecb-4328-a381-d6ecb4a7bcaf-iptables-alerter-script\") pod \"iptables-alerter-25rz7\" (UID: \"5560171c-fecb-4328-a381-d6ecb4a7bcaf\") " pod="openshift-network-operator/iptables-alerter-25rz7" Sep 09 23:06:53.680458 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680340 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/08bcba00-9881-442c-aa02-b875c99817c2-etc-tuned\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.680458 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680363 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-cnibin\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.680458 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680388 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-host-var-lib-cni-multus\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.680458 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680415 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3881b43c-4c39-4800-a4fd-a060fb8cf5e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pft9k\" (UID: \"3881b43c-4c39-4800-a4fd-a060fb8cf5e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" Sep 09 23:06:53.680458 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680442 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3881b43c-4c39-4800-a4fd-a060fb8cf5e8-registration-dir\") pod \"aws-ebs-csi-driver-node-pft9k\" (UID: \"3881b43c-4c39-4800-a4fd-a060fb8cf5e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" Sep 09 23:06:53.680853 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680468 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl8vx\" (UniqueName: \"kubernetes.io/projected/3881b43c-4c39-4800-a4fd-a060fb8cf5e8-kube-api-access-fl8vx\") pod \"aws-ebs-csi-driver-node-pft9k\" (UID: \"3881b43c-4c39-4800-a4fd-a060fb8cf5e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" Sep 09 23:06:53.680853 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680492 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-etc-sysctl-d\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.680853 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680515 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-run\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.680853 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680539 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a843f1d8-0cd1-4396-8fc1-5a22cd20f6e8-hosts-file\") pod \"node-resolver-jqxbd\" (UID: \"a843f1d8-0cd1-4396-8fc1-5a22cd20f6e8\") " pod="openshift-dns/node-resolver-jqxbd" Sep 09 23:06:53.680853 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680564 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-host-run-k8s-cni-cncf-io\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.680853 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680603 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-host-var-lib-cni-bin\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.680853 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680629 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-multus-conf-dir\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.680853 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680656 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxvn7\" (UniqueName: \"kubernetes.io/projected/5560171c-fecb-4328-a381-d6ecb4a7bcaf-kube-api-access-bxvn7\") pod \"iptables-alerter-25rz7\" (UID: \"5560171c-fecb-4328-a381-d6ecb4a7bcaf\") " pod="openshift-network-operator/iptables-alerter-25rz7" Sep 09 23:06:53.680853 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680680 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-etc-sysconfig\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.680853 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680704 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-var-lib-kubelet\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.680853 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680729 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-host\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.680853 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680758 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3881b43c-4c39-4800-a4fd-a060fb8cf5e8-socket-dir\") pod \"aws-ebs-csi-driver-node-pft9k\" (UID: \"3881b43c-4c39-4800-a4fd-a060fb8cf5e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" Sep 09 23:06:53.680853 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680788 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3881b43c-4c39-4800-a4fd-a060fb8cf5e8-device-dir\") pod \"aws-ebs-csi-driver-node-pft9k\" (UID: \"3881b43c-4c39-4800-a4fd-a060fb8cf5e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" Sep 09 23:06:53.680853 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680812 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-etc-kubernetes\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.680853 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680839 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-etc-sysctl-conf\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.681581 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680862 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-sys\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.681581 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680885 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-lib-modules\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.681581 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680936 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-system-cni-dir\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.681581 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680968 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-etc-kubernetes\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.681581 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.680994 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt58x\" (UniqueName: \"kubernetes.io/projected/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-kube-api-access-qt58x\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.681581 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.681019 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08bcba00-9881-442c-aa02-b875c99817c2-tmp\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.681581 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.681061 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l8d5\" (UniqueName: \"kubernetes.io/projected/a843f1d8-0cd1-4396-8fc1-5a22cd20f6e8-kube-api-access-6l8d5\") pod \"node-resolver-jqxbd\" (UID: \"a843f1d8-0cd1-4396-8fc1-5a22cd20f6e8\") " pod="openshift-dns/node-resolver-jqxbd" Sep 09 23:06:53.681581 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.681313 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z9xqn" Sep 09 23:06:53.682502 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.682244 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 09 23:06:53.682883 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.682863 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 09 23:06:53.683115 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.683098 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 09 23:06:53.683385 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.683350 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2c6gs\"" Sep 09 23:06:53.683632 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.683617 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 09 23:06:53.683860 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.683842 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 09 23:06:53.683958 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.683894 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 09 23:06:53.683958 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.683914 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 09 23:06:53.684074 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.683848 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 09 23:06:53.684227 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.684209 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hqcwb\"" Sep 09 23:06:53.684330 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.684316 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:06:53.684455 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:53.684428 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gw2dp" podUID="31b8a991-a7b8-4cae-9125-69fcd001f700" Sep 09 23:06:53.685903 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.685868 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 23:06:53.686904 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.686885 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:06:53.687011 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:53.686972 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c8p" podUID="a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd" Sep 09 23:06:53.689960 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.689642 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d76sb" Sep 09 23:06:53.693692 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.692989 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zncr7" Sep 09 23:06:53.693692 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.693006 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z6zvd\"" Sep 09 23:06:53.693692 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.693219 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 09 23:06:53.693692 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.693425 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 09 23:06:53.693692 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.693562 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 09 23:06:53.695581 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.695019 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:06:53.695581 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:53.695102 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j89rl" podUID="9e224f1b-9ce8-484e-8de5-f4f4f49a5eab" Sep 09 23:06:53.695581 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.695257 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 09 23:06:53.695581 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.695372 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wqdj6\"" Sep 09 23:06:53.695581 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.695256 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 09 23:06:53.709543 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.709508 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8sxmv" Sep 09 23:06:53.716505 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.716477 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8sxmv" Sep 09 23:06:53.770197 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.770165 2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 09 23:06:53.782029 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.781992 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-multus-cni-dir\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.782029 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782033 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-etc-modprobe-d\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.782275 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782063 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxc98\" (UniqueName: \"kubernetes.io/projected/08bcba00-9881-442c-aa02-b875c99817c2-kube-api-access-lxc98\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.782275 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782088 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a843f1d8-0cd1-4396-8fc1-5a22cd20f6e8-tmp-dir\") pod \"node-resolver-jqxbd\" (UID: \"a843f1d8-0cd1-4396-8fc1-5a22cd20f6e8\") " pod="openshift-dns/node-resolver-jqxbd" Sep 09 23:06:53.782275 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782115 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-systemd-units\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.782275 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782138 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-cni-binary-copy\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.782275 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782159 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d702c7a3-03a5-4b8b-9a7f-df843575cc43-serviceca\") pod \"node-ca-d76sb\" (UID: \"d702c7a3-03a5-4b8b-9a7f-df843575cc43\") " pod="openshift-image-registry/node-ca-d76sb" Sep 09 23:06:53.782275 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782161 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-multus-cni-dir\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.782275 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782181 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-host-run-netns\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.782275 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782225 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-etc-openvswitch\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.782275 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782228 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-etc-modprobe-d\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.782275 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782261 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-hostroot\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.782749 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782288 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/08bcba00-9881-442c-aa02-b875c99817c2-etc-tuned\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.782749 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782316 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-ovnkube-script-lib\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.782749 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782344 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-cnibin\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.782749 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782370 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3881b43c-4c39-4800-a4fd-a060fb8cf5e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pft9k\" (UID: \"3881b43c-4c39-4800-a4fd-a060fb8cf5e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" Sep 09 23:06:53.782749 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782393 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-run\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.782749 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782418 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a843f1d8-0cd1-4396-8fc1-5a22cd20f6e8-hosts-file\") pod \"node-resolver-jqxbd\" (UID: \"a843f1d8-0cd1-4396-8fc1-5a22cd20f6e8\") " pod="openshift-dns/node-resolver-jqxbd" Sep 09 23:06:53.782749 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782442 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-log-socket\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.782749 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782466 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-host-cni-netd\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.782749 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782490 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-env-overrides\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.782749 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782514 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b0c7b946-6758-4433-8081-5e3ea05b3639-cnibin\") pod \"multus-additional-cni-plugins-zncr7\" (UID: \"b0c7b946-6758-4433-8081-5e3ea05b3639\") " pod="openshift-multus/multus-additional-cni-plugins-zncr7" Sep 09 23:06:53.782749 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782537 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-multus-conf-dir\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.782749 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782569 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-multus-conf-dir\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.782749 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782590 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-run\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.782749 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782610 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a843f1d8-0cd1-4396-8fc1-5a22cd20f6e8-hosts-file\") pod \"node-resolver-jqxbd\" (UID: \"a843f1d8-0cd1-4396-8fc1-5a22cd20f6e8\") " pod="openshift-dns/node-resolver-jqxbd" Sep 09 23:06:53.782749 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782629 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-etc-sysconfig\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.782749 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782659 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs\") pod \"network-metrics-daemon-j89rl\" (UID: \"9e224f1b-9ce8-484e-8de5-f4f4f49a5eab\") " pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:06:53.782749 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782690 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-etc-kubernetes\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.783632 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782713 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-sys\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.783632 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782740 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-run-openvswitch\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.783632 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782764 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-etc-kubernetes\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.783632 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782792 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt58x\" (UniqueName: \"kubernetes.io/projected/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-kube-api-access-qt58x\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.783632 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782819 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08bcba00-9881-442c-aa02-b875c99817c2-tmp\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.783632 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782846 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b0c7b946-6758-4433-8081-5e3ea05b3639-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zncr7\" (UID: \"b0c7b946-6758-4433-8081-5e3ea05b3639\") " pod="openshift-multus/multus-additional-cni-plugins-zncr7" Sep 09 23:06:53.783632 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782875 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-host-kubelet\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.783632 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782903 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3881b43c-4c39-4800-a4fd-a060fb8cf5e8-sys-fs\") pod \"aws-ebs-csi-driver-node-pft9k\" (UID: \"3881b43c-4c39-4800-a4fd-a060fb8cf5e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" Sep 09 23:06:53.783632 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782926 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b0c7b946-6758-4433-8081-5e3ea05b3639-cni-binary-copy\") pod \"multus-additional-cni-plugins-zncr7\" (UID: \"b0c7b946-6758-4433-8081-5e3ea05b3639\") " pod="openshift-multus/multus-additional-cni-plugins-zncr7" Sep 09 23:06:53.783632 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782971 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49gm6\" (UniqueName: \"kubernetes.io/projected/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-kube-api-access-49gm6\") pod \"network-metrics-daemon-j89rl\" (UID: \"9e224f1b-9ce8-484e-8de5-f4f4f49a5eab\") " pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:06:53.783632 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.782996 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-host-cni-bin\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.783632 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783026 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31b8a991-a7b8-4cae-9125-69fcd001f700-original-pull-secret\") pod \"global-pull-secret-syncer-gw2dp\" (UID: \"31b8a991-a7b8-4cae-9125-69fcd001f700\") " pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:06:53.783632 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783052 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-etc-systemd\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.783632 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783107 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a843f1d8-0cd1-4396-8fc1-5a22cd20f6e8-tmp-dir\") pod \"node-resolver-jqxbd\" (UID: \"a843f1d8-0cd1-4396-8fc1-5a22cd20f6e8\") " pod="openshift-dns/node-resolver-jqxbd" Sep 09 23:06:53.783632 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783106 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-etc-kubernetes\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.783632 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783121 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b0c7b946-6758-4433-8081-5e3ea05b3639-system-cni-dir\") pod \"multus-additional-cni-plugins-zncr7\" (UID: \"b0c7b946-6758-4433-8081-5e3ea05b3639\") " pod="openshift-multus/multus-additional-cni-plugins-zncr7" Sep 09 23:06:53.783632 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783170 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps98r\" (UniqueName: \"kubernetes.io/projected/b0c7b946-6758-4433-8081-5e3ea05b3639-kube-api-access-ps98r\") pod \"multus-additional-cni-plugins-zncr7\" (UID: \"b0c7b946-6758-4433-8081-5e3ea05b3639\") " pod="openshift-multus/multus-additional-cni-plugins-zncr7" Sep 09 23:06:53.784412 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783198 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3881b43c-4c39-4800-a4fd-a060fb8cf5e8-sys-fs\") pod \"aws-ebs-csi-driver-node-pft9k\" (UID: \"3881b43c-4c39-4800-a4fd-a060fb8cf5e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" Sep 09 23:06:53.784412 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783237 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.784412 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783259 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-host-var-lib-kubelet\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.784412 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783261 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-cnibin\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.784412 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783257 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-cni-binary-copy\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.784412 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783277 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-multus-daemon-config\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.784412 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783295 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5560171c-fecb-4328-a381-d6ecb4a7bcaf-host-slash\") pod \"iptables-alerter-25rz7\" (UID: \"5560171c-fecb-4328-a381-d6ecb4a7bcaf\") " pod="openshift-network-operator/iptables-alerter-25rz7" Sep 09 23:06:53.784412 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783303 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-sys\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.784412 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783312 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t78d5\" (UniqueName: \"kubernetes.io/projected/a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd-kube-api-access-t78d5\") pod \"network-check-target-v7c8p\" (UID: \"a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd\") " pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:06:53.784412 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783353 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-multus-socket-dir-parent\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.784412 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783367 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-hostroot\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.784412 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783373 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-host-run-netns\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.784412 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783386 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-etc-sysconfig\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.784412 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783390 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5560171c-fecb-4328-a381-d6ecb4a7bcaf-iptables-alerter-script\") pod \"iptables-alerter-25rz7\" (UID: \"5560171c-fecb-4328-a381-d6ecb4a7bcaf\") " pod="openshift-network-operator/iptables-alerter-25rz7" Sep 09 23:06:53.784412 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783410 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3881b43c-4c39-4800-a4fd-a060fb8cf5e8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pft9k\" (UID: \"3881b43c-4c39-4800-a4fd-a060fb8cf5e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" Sep 09 23:06:53.784412 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783428 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-host-run-ovn-kubernetes\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.784412 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783432 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-etc-kubernetes\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.785210 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783463 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-host-run-netns\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.785210 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783452 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-etc-systemd\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.785210 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783487 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5560171c-fecb-4328-a381-d6ecb4a7bcaf-host-slash\") pod \"iptables-alerter-25rz7\" (UID: \"5560171c-fecb-4328-a381-d6ecb4a7bcaf\") " pod="openshift-network-operator/iptables-alerter-25rz7" Sep 09 23:06:53.785210 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783493 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-host-var-lib-kubelet\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.785210 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783517 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d5402f8c-eefb-4fb3-9742-b96b8207198f-agent-certs\") pod \"konnectivity-agent-z9xqn\" (UID: \"d5402f8c-eefb-4fb3-9742-b96b8207198f\") " pod="kube-system/konnectivity-agent-z9xqn" Sep 09 23:06:53.785210 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783549 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-host-var-lib-cni-multus\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.785210 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783582 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-host-var-lib-cni-multus\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.785210 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783594 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3881b43c-4c39-4800-a4fd-a060fb8cf5e8-registration-dir\") pod \"aws-ebs-csi-driver-node-pft9k\" (UID: \"3881b43c-4c39-4800-a4fd-a060fb8cf5e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" Sep 09 23:06:53.785210 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783614 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-multus-socket-dir-parent\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.785210 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783636 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl8vx\" (UniqueName: \"kubernetes.io/projected/3881b43c-4c39-4800-a4fd-a060fb8cf5e8-kube-api-access-fl8vx\") pod \"aws-ebs-csi-driver-node-pft9k\" (UID: \"3881b43c-4c39-4800-a4fd-a060fb8cf5e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" Sep 09 23:06:53.785210 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783663 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3881b43c-4c39-4800-a4fd-a060fb8cf5e8-registration-dir\") pod \"aws-ebs-csi-driver-node-pft9k\" (UID: \"3881b43c-4c39-4800-a4fd-a060fb8cf5e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" Sep 09 23:06:53.785210 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783707 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-etc-sysctl-d\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.785210 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783750 2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 09 23:06:53.785210 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783820 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d702c7a3-03a5-4b8b-9a7f-df843575cc43-host\") pod \"node-ca-d76sb\" (UID: \"d702c7a3-03a5-4b8b-9a7f-df843575cc43\") " pod="openshift-image-registry/node-ca-d76sb" Sep 09 23:06:53.785210 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783888 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-node-log\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.785210 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783904 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5560171c-fecb-4328-a381-d6ecb4a7bcaf-iptables-alerter-script\") pod \"iptables-alerter-25rz7\" (UID: \"5560171c-fecb-4328-a381-d6ecb4a7bcaf\") " pod="openshift-network-operator/iptables-alerter-25rz7" Sep 09 23:06:53.785210 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783924 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/31b8a991-a7b8-4cae-9125-69fcd001f700-kubelet-config\") pod \"global-pull-secret-syncer-gw2dp\" (UID: \"31b8a991-a7b8-4cae-9125-69fcd001f700\") " pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:06:53.786223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783949 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/31b8a991-a7b8-4cae-9125-69fcd001f700-dbus\") pod \"global-pull-secret-syncer-gw2dp\" (UID: \"31b8a991-a7b8-4cae-9125-69fcd001f700\") " pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:06:53.786223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783956 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-multus-daemon-config\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.786223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.783979 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-etc-sysctl-d\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.786223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784009 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjjlz\" (UniqueName: \"kubernetes.io/projected/d702c7a3-03a5-4b8b-9a7f-df843575cc43-kube-api-access-rjjlz\") pod \"node-ca-d76sb\" (UID: \"d702c7a3-03a5-4b8b-9a7f-df843575cc43\") " pod="openshift-image-registry/node-ca-d76sb" Sep 09 23:06:53.786223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784045 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-host-run-k8s-cni-cncf-io\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.786223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784079 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-host-var-lib-cni-bin\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.786223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784095 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-host-run-k8s-cni-cncf-io\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.786223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784122 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxvn7\" (UniqueName: \"kubernetes.io/projected/5560171c-fecb-4328-a381-d6ecb4a7bcaf-kube-api-access-bxvn7\") pod \"iptables-alerter-25rz7\" (UID: \"5560171c-fecb-4328-a381-d6ecb4a7bcaf\") " pod="openshift-network-operator/iptables-alerter-25rz7" Sep 09 23:06:53.786223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784147 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-var-lib-kubelet\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.786223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784200 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-var-lib-kubelet\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.786223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784199 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-host-var-lib-cni-bin\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.786223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784268 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-host\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.786223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784296 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-run-ovn\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.786223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784319 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-ovn-node-metrics-cert\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.786223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784344 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-host\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.786223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784344 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3881b43c-4c39-4800-a4fd-a060fb8cf5e8-socket-dir\") pod \"aws-ebs-csi-driver-node-pft9k\" (UID: \"3881b43c-4c39-4800-a4fd-a060fb8cf5e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" Sep 09 23:06:53.786223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784388 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3881b43c-4c39-4800-a4fd-a060fb8cf5e8-device-dir\") pod \"aws-ebs-csi-driver-node-pft9k\" (UID: \"3881b43c-4c39-4800-a4fd-a060fb8cf5e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" Sep 09 23:06:53.787114 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784414 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-etc-sysctl-conf\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.787114 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784434 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3881b43c-4c39-4800-a4fd-a060fb8cf5e8-socket-dir\") pod \"aws-ebs-csi-driver-node-pft9k\" (UID: \"3881b43c-4c39-4800-a4fd-a060fb8cf5e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" Sep 09 23:06:53.787114 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784475 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-lib-modules\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.787114 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784485 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3881b43c-4c39-4800-a4fd-a060fb8cf5e8-device-dir\") pod \"aws-ebs-csi-driver-node-pft9k\" (UID: \"3881b43c-4c39-4800-a4fd-a060fb8cf5e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" Sep 09 23:06:53.787114 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784501 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l8d5\" (UniqueName: \"kubernetes.io/projected/a843f1d8-0cd1-4396-8fc1-5a22cd20f6e8-kube-api-access-6l8d5\") pod \"node-resolver-jqxbd\" (UID: \"a843f1d8-0cd1-4396-8fc1-5a22cd20f6e8\") " pod="openshift-dns/node-resolver-jqxbd" Sep 09 23:06:53.787114 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784530 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-host-slash\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.787114 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784556 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-var-lib-openvswitch\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.787114 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784582 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-system-cni-dir\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.787114 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784608 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b0c7b946-6758-4433-8081-5e3ea05b3639-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zncr7\" (UID: \"b0c7b946-6758-4433-8081-5e3ea05b3639\") " pod="openshift-multus/multus-additional-cni-plugins-zncr7" Sep 09 23:06:53.787114 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784663 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b0c7b946-6758-4433-8081-5e3ea05b3639-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zncr7\" (UID: \"b0c7b946-6758-4433-8081-5e3ea05b3639\") " pod="openshift-multus/multus-additional-cni-plugins-zncr7" Sep 09 23:06:53.787114 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784664 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-lib-modules\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.787114 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784582 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/08bcba00-9881-442c-aa02-b875c99817c2-etc-sysctl-conf\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.787114 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784690 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-run-systemd\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.787114 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784715 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-ovnkube-config\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.787114 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784731 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sqc6\" (UniqueName: \"kubernetes.io/projected/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-kube-api-access-6sqc6\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.787114 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784751 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-host-run-multus-certs\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.787114 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784731 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-system-cni-dir\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.788038 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784780 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-host-run-multus-certs\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.788038 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784783 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3881b43c-4c39-4800-a4fd-a060fb8cf5e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-pft9k\" (UID: \"3881b43c-4c39-4800-a4fd-a060fb8cf5e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" Sep 09 23:06:53.788038 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784805 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d5402f8c-eefb-4fb3-9742-b96b8207198f-konnectivity-ca\") pod \"konnectivity-agent-z9xqn\" (UID: \"d5402f8c-eefb-4fb3-9742-b96b8207198f\") " pod="kube-system/konnectivity-agent-z9xqn" Sep 09 23:06:53.788038 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.784848 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3881b43c-4c39-4800-a4fd-a060fb8cf5e8-etc-selinux\") pod \"aws-ebs-csi-driver-node-pft9k\" (UID: \"3881b43c-4c39-4800-a4fd-a060fb8cf5e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" Sep 09 23:06:53.788327 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.788104 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08bcba00-9881-442c-aa02-b875c99817c2-tmp\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.788327 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.788151 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/08bcba00-9881-442c-aa02-b875c99817c2-etc-tuned\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.791188 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.791087 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxc98\" (UniqueName: \"kubernetes.io/projected/08bcba00-9881-442c-aa02-b875c99817c2-kube-api-access-lxc98\") pod \"tuned-5qpkc\" (UID: \"08bcba00-9881-442c-aa02-b875c99817c2\") " pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:53.792071 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.792023 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt58x\" (UniqueName: \"kubernetes.io/projected/1c6b309f-307c-4a0f-a0d1-586fadcd13b1-kube-api-access-qt58x\") pod \"multus-t69sp\" (UID: \"1c6b309f-307c-4a0f-a0d1-586fadcd13b1\") " pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.792991 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.792919 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl8vx\" (UniqueName: \"kubernetes.io/projected/3881b43c-4c39-4800-a4fd-a060fb8cf5e8-kube-api-access-fl8vx\") pod \"aws-ebs-csi-driver-node-pft9k\" (UID: \"3881b43c-4c39-4800-a4fd-a060fb8cf5e8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" Sep 09 23:06:53.793125 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.793096 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxvn7\" (UniqueName: \"kubernetes.io/projected/5560171c-fecb-4328-a381-d6ecb4a7bcaf-kube-api-access-bxvn7\") pod \"iptables-alerter-25rz7\" (UID: \"5560171c-fecb-4328-a381-d6ecb4a7bcaf\") " pod="openshift-network-operator/iptables-alerter-25rz7" Sep 09 23:06:53.794081 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.793777 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l8d5\" (UniqueName: \"kubernetes.io/projected/a843f1d8-0cd1-4396-8fc1-5a22cd20f6e8-kube-api-access-6l8d5\") pod \"node-resolver-jqxbd\" (UID: \"a843f1d8-0cd1-4396-8fc1-5a22cd20f6e8\") " pod="openshift-dns/node-resolver-jqxbd" Sep 09 23:06:53.808423 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.808349 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" event={"ID":"47e768c654fbb38de18e4644c0ab2928","Type":"ContainerStarted","Data":"906cb28ce07c5bfbe0dd4f1948966c1e38b8d705502f188499421494106039c2"} Sep 09 23:06:53.809994 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.809959 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-192.ec2.internal" event={"ID":"ef36ec20e9fe178e1d7d8c3ae65f1ea2","Type":"ContainerStarted","Data":"65bffe6fbf2a292dd39c5eaa4e442801aa82dc51ba9220c5e2ff7119c2517436"} Sep 09 23:06:53.882791 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.882755 2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 23:06:53.885920 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.885893 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-log-socket\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.885920 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.885944 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-host-cni-netd\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.886170 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.885981 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-env-overrides\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.886170 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886012 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b0c7b946-6758-4433-8081-5e3ea05b3639-cnibin\") pod \"multus-additional-cni-plugins-zncr7\" (UID: \"b0c7b946-6758-4433-8081-5e3ea05b3639\") " pod="openshift-multus/multus-additional-cni-plugins-zncr7" Sep 09 23:06:53.886170 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886023 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-log-socket\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.886170 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886042 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs\") pod \"network-metrics-daemon-j89rl\" (UID: \"9e224f1b-9ce8-484e-8de5-f4f4f49a5eab\") " pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:06:53.886170 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886047 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-host-cni-netd\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.886170 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886091 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b0c7b946-6758-4433-8081-5e3ea05b3639-cnibin\") pod \"multus-additional-cni-plugins-zncr7\" (UID: \"b0c7b946-6758-4433-8081-5e3ea05b3639\") " pod="openshift-multus/multus-additional-cni-plugins-zncr7" Sep 09 23:06:53.886170 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886122 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-run-openvswitch\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.886170 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886151 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b0c7b946-6758-4433-8081-5e3ea05b3639-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zncr7\" (UID: \"b0c7b946-6758-4433-8081-5e3ea05b3639\") " pod="openshift-multus/multus-additional-cni-plugins-zncr7" Sep 09 23:06:53.886170 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:53.886166 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:06:53.886585 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886178 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-host-kubelet\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.886585 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886191 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-run-openvswitch\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.886585 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:53.886251 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs podName:9e224f1b-9ce8-484e-8de5-f4f4f49a5eab nodeName:}" failed. No retries permitted until 2026-09-09 23:06:54.386226808 +0000 UTC m=+3.072010216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs") pod "network-metrics-daemon-j89rl" (UID: "9e224f1b-9ce8-484e-8de5-f4f4f49a5eab") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:06:53.886585 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886207 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b0c7b946-6758-4433-8081-5e3ea05b3639-cni-binary-copy\") pod \"multus-additional-cni-plugins-zncr7\" (UID: \"b0c7b946-6758-4433-8081-5e3ea05b3639\") " pod="openshift-multus/multus-additional-cni-plugins-zncr7" Sep 09 23:06:53.886585 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886251 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-host-kubelet\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.886585 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886283 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49gm6\" (UniqueName: \"kubernetes.io/projected/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-kube-api-access-49gm6\") pod \"network-metrics-daemon-j89rl\" (UID: \"9e224f1b-9ce8-484e-8de5-f4f4f49a5eab\") " pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:06:53.886585 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886309 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-host-cni-bin\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.886585 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886336 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31b8a991-a7b8-4cae-9125-69fcd001f700-original-pull-secret\") pod \"global-pull-secret-syncer-gw2dp\" (UID: \"31b8a991-a7b8-4cae-9125-69fcd001f700\") " pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:06:53.886585 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886407 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b0c7b946-6758-4433-8081-5e3ea05b3639-system-cni-dir\") pod \"multus-additional-cni-plugins-zncr7\" (UID: \"b0c7b946-6758-4433-8081-5e3ea05b3639\") " pod="openshift-multus/multus-additional-cni-plugins-zncr7" Sep 09 23:06:53.886585 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886434 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps98r\" (UniqueName: \"kubernetes.io/projected/b0c7b946-6758-4433-8081-5e3ea05b3639-kube-api-access-ps98r\") pod \"multus-additional-cni-plugins-zncr7\" (UID: \"b0c7b946-6758-4433-8081-5e3ea05b3639\") " pod="openshift-multus/multus-additional-cni-plugins-zncr7" Sep 09 23:06:53.886585 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886461 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.886585 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886510 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t78d5\" (UniqueName: \"kubernetes.io/projected/a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd-kube-api-access-t78d5\") pod \"network-check-target-v7c8p\" (UID: \"a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd\") " pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:06:53.886585 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886517 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-env-overrides\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.886585 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886539 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-host-run-ovn-kubernetes\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.886585 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:53.886558 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:06:53.886585 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886575 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d5402f8c-eefb-4fb3-9742-b96b8207198f-agent-certs\") pod \"konnectivity-agent-z9xqn\" (UID: \"d5402f8c-eefb-4fb3-9742-b96b8207198f\") " pod="kube-system/konnectivity-agent-z9xqn" Sep 09 23:06:53.887466 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:53.886607 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31b8a991-a7b8-4cae-9125-69fcd001f700-original-pull-secret podName:31b8a991-a7b8-4cae-9125-69fcd001f700 nodeName:}" failed. No retries permitted until 2026-09-09 23:06:54.386591486 +0000 UTC m=+3.072374884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31b8a991-a7b8-4cae-9125-69fcd001f700-original-pull-secret") pod "global-pull-secret-syncer-gw2dp" (UID: "31b8a991-a7b8-4cae-9125-69fcd001f700") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:06:53.887466 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886627 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d702c7a3-03a5-4b8b-9a7f-df843575cc43-host\") pod \"node-ca-d76sb\" (UID: \"d702c7a3-03a5-4b8b-9a7f-df843575cc43\") " pod="openshift-image-registry/node-ca-d76sb" Sep 09 23:06:53.887466 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886655 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-node-log\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.887466 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886687 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/31b8a991-a7b8-4cae-9125-69fcd001f700-kubelet-config\") pod \"global-pull-secret-syncer-gw2dp\" (UID: \"31b8a991-a7b8-4cae-9125-69fcd001f700\") " pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:06:53.887466 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886715 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/31b8a991-a7b8-4cae-9125-69fcd001f700-dbus\") pod \"global-pull-secret-syncer-gw2dp\" (UID: \"31b8a991-a7b8-4cae-9125-69fcd001f700\") " pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:06:53.887466 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886740 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjjlz\" (UniqueName: \"kubernetes.io/projected/d702c7a3-03a5-4b8b-9a7f-df843575cc43-kube-api-access-rjjlz\") pod \"node-ca-d76sb\" (UID: \"d702c7a3-03a5-4b8b-9a7f-df843575cc43\") " pod="openshift-image-registry/node-ca-d76sb" Sep 09 23:06:53.887466 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886798 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-run-ovn\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.887466 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886800 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b0c7b946-6758-4433-8081-5e3ea05b3639-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zncr7\" (UID: \"b0c7b946-6758-4433-8081-5e3ea05b3639\") " pod="openshift-multus/multus-additional-cni-plugins-zncr7" Sep 09 23:06:53.887466 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886826 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-ovn-node-metrics-cert\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.887466 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886846 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b0c7b946-6758-4433-8081-5e3ea05b3639-cni-binary-copy\") pod \"multus-additional-cni-plugins-zncr7\" (UID: \"b0c7b946-6758-4433-8081-5e3ea05b3639\") " pod="openshift-multus/multus-additional-cni-plugins-zncr7" Sep 09 23:06:53.887466 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886872 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-host-slash\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.887466 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886873 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.887466 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886898 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-var-lib-openvswitch\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.887466 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.886944 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b0c7b946-6758-4433-8081-5e3ea05b3639-system-cni-dir\") pod \"multus-additional-cni-plugins-zncr7\" (UID: \"b0c7b946-6758-4433-8081-5e3ea05b3639\") " pod="openshift-multus/multus-additional-cni-plugins-zncr7" Sep 09 23:06:53.887466 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887038 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/31b8a991-a7b8-4cae-9125-69fcd001f700-dbus\") pod \"global-pull-secret-syncer-gw2dp\" (UID: \"31b8a991-a7b8-4cae-9125-69fcd001f700\") " pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:06:53.887466 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887064 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b0c7b946-6758-4433-8081-5e3ea05b3639-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zncr7\" (UID: \"b0c7b946-6758-4433-8081-5e3ea05b3639\") " pod="openshift-multus/multus-additional-cni-plugins-zncr7" Sep 09 23:06:53.888311 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887097 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-var-lib-openvswitch\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.888311 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887102 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b0c7b946-6758-4433-8081-5e3ea05b3639-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zncr7\" (UID: \"b0c7b946-6758-4433-8081-5e3ea05b3639\") " pod="openshift-multus/multus-additional-cni-plugins-zncr7" Sep 09 23:06:53.888311 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887099 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-node-log\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.888311 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887204 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-run-systemd\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.888311 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887240 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-ovnkube-config\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.888311 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887259 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b0c7b946-6758-4433-8081-5e3ea05b3639-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zncr7\" (UID: \"b0c7b946-6758-4433-8081-5e3ea05b3639\") " pod="openshift-multus/multus-additional-cni-plugins-zncr7" Sep 09 23:06:53.888311 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887267 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sqc6\" (UniqueName: \"kubernetes.io/projected/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-kube-api-access-6sqc6\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.888311 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887316 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d5402f8c-eefb-4fb3-9742-b96b8207198f-konnectivity-ca\") pod \"konnectivity-agent-z9xqn\" (UID: \"d5402f8c-eefb-4fb3-9742-b96b8207198f\") " pod="kube-system/konnectivity-agent-z9xqn" Sep 09 23:06:53.888311 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887361 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-systemd-units\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.888311 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887389 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d702c7a3-03a5-4b8b-9a7f-df843575cc43-serviceca\") pod \"node-ca-d76sb\" (UID: \"d702c7a3-03a5-4b8b-9a7f-df843575cc43\") " pod="openshift-image-registry/node-ca-d76sb" Sep 09 23:06:53.888311 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887415 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-host-run-netns\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.888311 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887427 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d702c7a3-03a5-4b8b-9a7f-df843575cc43-host\") pod \"node-ca-d76sb\" (UID: \"d702c7a3-03a5-4b8b-9a7f-df843575cc43\") " pod="openshift-image-registry/node-ca-d76sb" Sep 09 23:06:53.888311 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887440 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-etc-openvswitch\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.888311 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887500 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/31b8a991-a7b8-4cae-9125-69fcd001f700-kubelet-config\") pod \"global-pull-secret-syncer-gw2dp\" (UID: \"31b8a991-a7b8-4cae-9125-69fcd001f700\") " pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:06:53.888311 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887521 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-run-ovn\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.888311 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887546 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-host-run-ovn-kubernetes\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.888311 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887549 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-systemd-units\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.889198 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887577 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-host-run-netns\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.889198 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887594 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-host-slash\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.889198 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887602 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b0c7b946-6758-4433-8081-5e3ea05b3639-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zncr7\" (UID: \"b0c7b946-6758-4433-8081-5e3ea05b3639\") " pod="openshift-multus/multus-additional-cni-plugins-zncr7" Sep 09 23:06:53.889198 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887608 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-etc-openvswitch\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.889198 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887676 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-host-cni-bin\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.889198 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887715 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-ovnkube-script-lib\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.889198 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887893 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-run-systemd\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.889198 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.887944 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-ovnkube-config\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.889198 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.888139 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d702c7a3-03a5-4b8b-9a7f-df843575cc43-serviceca\") pod \"node-ca-d76sb\" (UID: \"d702c7a3-03a5-4b8b-9a7f-df843575cc43\") " pod="openshift-image-registry/node-ca-d76sb" Sep 09 23:06:53.889198 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.888249 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d5402f8c-eefb-4fb3-9742-b96b8207198f-konnectivity-ca\") pod \"konnectivity-agent-z9xqn\" (UID: \"d5402f8c-eefb-4fb3-9742-b96b8207198f\") " pod="kube-system/konnectivity-agent-z9xqn" Sep 09 23:06:53.889198 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.888371 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-ovnkube-script-lib\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.891194 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.891171 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-ovn-node-metrics-cert\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.891422 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.891404 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d5402f8c-eefb-4fb3-9742-b96b8207198f-agent-certs\") pod \"konnectivity-agent-z9xqn\" (UID: \"d5402f8c-eefb-4fb3-9742-b96b8207198f\") " pod="kube-system/konnectivity-agent-z9xqn" Sep 09 23:06:53.897063 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:53.897032 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:06:53.897063 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:53.897065 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:06:53.897228 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:53.897082 2561 projected.go:196] Error preparing data for projected volume kube-api-access-t78d5 for pod openshift-network-diagnostics/network-check-target-v7c8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:06:53.897228 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:53.897158 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd-kube-api-access-t78d5 podName:a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd nodeName:}" failed. No retries permitted until 2026-09-09 23:06:54.397133659 +0000 UTC m=+3.082917057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t78d5" (UniqueName: "kubernetes.io/projected/a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd-kube-api-access-t78d5") pod "network-check-target-v7c8p" (UID: "a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:06:53.899304 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.899260 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjjlz\" (UniqueName: \"kubernetes.io/projected/d702c7a3-03a5-4b8b-9a7f-df843575cc43-kube-api-access-rjjlz\") pod \"node-ca-d76sb\" (UID: \"d702c7a3-03a5-4b8b-9a7f-df843575cc43\") " pod="openshift-image-registry/node-ca-d76sb" Sep 09 23:06:53.900318 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.900247 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49gm6\" (UniqueName: \"kubernetes.io/projected/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-kube-api-access-49gm6\") pod \"network-metrics-daemon-j89rl\" (UID: \"9e224f1b-9ce8-484e-8de5-f4f4f49a5eab\") " pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:06:53.901839 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.901813 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps98r\" (UniqueName: \"kubernetes.io/projected/b0c7b946-6758-4433-8081-5e3ea05b3639-kube-api-access-ps98r\") pod \"multus-additional-cni-plugins-zncr7\" (UID: \"b0c7b946-6758-4433-8081-5e3ea05b3639\") " pod="openshift-multus/multus-additional-cni-plugins-zncr7" Sep 09 23:06:53.904115 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.904087 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sqc6\" (UniqueName: \"kubernetes.io/projected/6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21-kube-api-access-6sqc6\") pod \"ovnkube-node-pbrgn\" (UID: \"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21\") " pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:53.915110 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.915066 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 23:06:53.987906 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.987856 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t69sp" Sep 09 23:06:53.998122 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:53.998089 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" Sep 09 23:06:54.007468 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:54.007434 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" Sep 09 23:06:54.015871 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:54.015832 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-25rz7" Sep 09 23:06:54.024081 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:54.024032 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jqxbd" Sep 09 23:06:54.034452 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:54.034406 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:06:54.041651 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:54.041628 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z9xqn" Sep 09 23:06:54.050934 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:54.050895 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d76sb" Sep 09 23:06:54.055932 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:54.055906 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zncr7" Sep 09 23:06:54.393340 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:54.393240 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs\") pod \"network-metrics-daemon-j89rl\" (UID: \"9e224f1b-9ce8-484e-8de5-f4f4f49a5eab\") " pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:06:54.393340 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:54.393303 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31b8a991-a7b8-4cae-9125-69fcd001f700-original-pull-secret\") pod \"global-pull-secret-syncer-gw2dp\" (UID: \"31b8a991-a7b8-4cae-9125-69fcd001f700\") " pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:06:54.393578 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:54.393413 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:06:54.393578 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:54.393417 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:06:54.393578 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:54.393487 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31b8a991-a7b8-4cae-9125-69fcd001f700-original-pull-secret podName:31b8a991-a7b8-4cae-9125-69fcd001f700 nodeName:}" failed. No retries permitted until 2026-09-09 23:06:55.393465712 +0000 UTC m=+4.079249095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31b8a991-a7b8-4cae-9125-69fcd001f700-original-pull-secret") pod "global-pull-secret-syncer-gw2dp" (UID: "31b8a991-a7b8-4cae-9125-69fcd001f700") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:06:54.393578 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:54.393509 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs podName:9e224f1b-9ce8-484e-8de5-f4f4f49a5eab nodeName:}" failed. No retries permitted until 2026-09-09 23:06:55.393498566 +0000 UTC m=+4.079281952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs") pod "network-metrics-daemon-j89rl" (UID: "9e224f1b-9ce8-484e-8de5-f4f4f49a5eab") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:06:54.494202 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:54.494167 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t78d5\" (UniqueName: \"kubernetes.io/projected/a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd-kube-api-access-t78d5\") pod \"network-check-target-v7c8p\" (UID: \"a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd\") " pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:06:54.494423 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:54.494308 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:06:54.494423 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:54.494334 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:06:54.494423 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:54.494349 2561 projected.go:196] Error preparing data for projected volume kube-api-access-t78d5 for pod openshift-network-diagnostics/network-check-target-v7c8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:06:54.494423 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:54.494404 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd-kube-api-access-t78d5 podName:a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd nodeName:}" failed. No retries permitted until 2026-09-09 23:06:55.494390603 +0000 UTC m=+4.180173985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t78d5" (UniqueName: "kubernetes.io/projected/a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd-kube-api-access-t78d5") pod "network-check-target-v7c8p" (UID: "a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:06:54.720064 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:54.719923 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 23:01:53 +0000 UTC" deadline="2028-03-05 21:53:57.397268541 +0000 UTC" Sep 09 23:06:54.720064 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:54.719994 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13030h47m2.677278956s" Sep 09 23:06:54.972070 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:54.972037 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08bcba00_9881_442c_aa02_b875c99817c2.slice/crio-d4e6fbb16473a0184b747a1b0c0c5bb2b1fbefe5d95dfe3a60766881cfb2aa83 WatchSource:0}: Error finding container d4e6fbb16473a0184b747a1b0c0c5bb2b1fbefe5d95dfe3a60766881cfb2aa83: Status 404 returned error can't find the container with id d4e6fbb16473a0184b747a1b0c0c5bb2b1fbefe5d95dfe3a60766881cfb2aa83 Sep 09 23:06:54.974132 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:54.974104 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5402f8c_eefb_4fb3_9742_b96b8207198f.slice/crio-54cde1c0629d306f6c436195820ca7ba0cdb78aad7fe6f7eeace9b46c498e1a6 WatchSource:0}: Error finding container 54cde1c0629d306f6c436195820ca7ba0cdb78aad7fe6f7eeace9b46c498e1a6: Status 404 returned error can't find the container with id 54cde1c0629d306f6c436195820ca7ba0cdb78aad7fe6f7eeace9b46c498e1a6 Sep 09 23:06:54.975699 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:54.975672 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c7b946_6758_4433_8081_5e3ea05b3639.slice/crio-88fa022a497fc6b9ef88375a88c7fd4d0996f3efb7b38c1e700b918f03c29ff3 WatchSource:0}: Error finding container 88fa022a497fc6b9ef88375a88c7fd4d0996f3efb7b38c1e700b918f03c29ff3: Status 404 returned error can't find the container with id 88fa022a497fc6b9ef88375a88c7fd4d0996f3efb7b38c1e700b918f03c29ff3 Sep 09 23:06:54.978370 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:06:54.978342 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c6b309f_307c_4a0f_a0d1_586fadcd13b1.slice/crio-aefc80c59c8df8029d382724a9e787f30e1b74bcad8d2c39858847b4622133f1 WatchSource:0}: Error finding container aefc80c59c8df8029d382724a9e787f30e1b74bcad8d2c39858847b4622133f1: Status 404 returned error can't find the container with id aefc80c59c8df8029d382724a9e787f30e1b74bcad8d2c39858847b4622133f1 Sep 09 23:06:55.400654 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:55.400610 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs\") pod \"network-metrics-daemon-j89rl\" (UID: \"9e224f1b-9ce8-484e-8de5-f4f4f49a5eab\") " pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:06:55.400845 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:55.400662 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31b8a991-a7b8-4cae-9125-69fcd001f700-original-pull-secret\") pod \"global-pull-secret-syncer-gw2dp\" (UID: \"31b8a991-a7b8-4cae-9125-69fcd001f700\") " pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:06:55.400845 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:55.400790 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:06:55.400845 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:55.400793 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:06:55.401097 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:55.400856 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31b8a991-a7b8-4cae-9125-69fcd001f700-original-pull-secret podName:31b8a991-a7b8-4cae-9125-69fcd001f700 nodeName:}" failed. No retries permitted until 2026-09-09 23:06:57.400841168 +0000 UTC m=+6.086624550 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31b8a991-a7b8-4cae-9125-69fcd001f700-original-pull-secret") pod "global-pull-secret-syncer-gw2dp" (UID: "31b8a991-a7b8-4cae-9125-69fcd001f700") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:06:55.401097 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:55.400877 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs podName:9e224f1b-9ce8-484e-8de5-f4f4f49a5eab nodeName:}" failed. No retries permitted until 2026-09-09 23:06:57.400868251 +0000 UTC m=+6.086651633 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs") pod "network-metrics-daemon-j89rl" (UID: "9e224f1b-9ce8-484e-8de5-f4f4f49a5eab") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:06:55.501852 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:55.501749 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t78d5\" (UniqueName: \"kubernetes.io/projected/a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd-kube-api-access-t78d5\") pod \"network-check-target-v7c8p\" (UID: \"a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd\") " pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:06:55.502063 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:55.502000 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:06:55.502063 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:55.502024 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:06:55.502063 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:55.502039 2561 projected.go:196] Error preparing data for projected volume kube-api-access-t78d5 for pod openshift-network-diagnostics/network-check-target-v7c8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:06:55.502214 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:55.502107 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd-kube-api-access-t78d5 podName:a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd nodeName:}" failed. No retries permitted until 2026-09-09 23:06:57.502086282 +0000 UTC m=+6.187869672 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t78d5" (UniqueName: "kubernetes.io/projected/a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd-kube-api-access-t78d5") pod "network-check-target-v7c8p" (UID: "a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:06:55.724952 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:55.721031 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 23:01:53 +0000 UTC" deadline="2028-05-28 08:00:47.228563478 +0000 UTC" Sep 09 23:06:55.724952 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:55.721073 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15032h53m51.507495256s" Sep 09 23:06:55.807727 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:55.806699 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:06:55.807727 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:55.806858 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gw2dp" podUID="31b8a991-a7b8-4cae-9125-69fcd001f700" Sep 09 23:06:55.807727 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:55.807318 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:06:55.807727 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:55.807420 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c8p" podUID="a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd" Sep 09 23:06:55.807727 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:55.807496 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:06:55.807727 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:55.807590 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j89rl" podUID="9e224f1b-9ce8-484e-8de5-f4f4f49a5eab" Sep 09 23:06:55.835070 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:55.835030 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zncr7" event={"ID":"b0c7b946-6758-4433-8081-5e3ea05b3639","Type":"ContainerStarted","Data":"88fa022a497fc6b9ef88375a88c7fd4d0996f3efb7b38c1e700b918f03c29ff3"} Sep 09 23:06:55.840832 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:55.840759 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z9xqn" event={"ID":"d5402f8c-eefb-4fb3-9742-b96b8207198f","Type":"ContainerStarted","Data":"54cde1c0629d306f6c436195820ca7ba0cdb78aad7fe6f7eeace9b46c498e1a6"} Sep 09 23:06:55.851488 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:55.851445 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" event={"ID":"08bcba00-9881-442c-aa02-b875c99817c2","Type":"ContainerStarted","Data":"d4e6fbb16473a0184b747a1b0c0c5bb2b1fbefe5d95dfe3a60766881cfb2aa83"} Sep 09 23:06:55.864430 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:55.864382 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" event={"ID":"3881b43c-4c39-4800-a4fd-a060fb8cf5e8","Type":"ContainerStarted","Data":"3582ce75f31e5ad5fd738cec595c654ddb84cdefa2646f3c7d1729ef7d0969c9"} Sep 09 23:06:55.870749 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:55.870707 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jqxbd" event={"ID":"a843f1d8-0cd1-4396-8fc1-5a22cd20f6e8","Type":"ContainerStarted","Data":"f9f862ad4584d07b09bbc9319a6cd578545b52866f76fc50bcbff8c0518825d8"} Sep 09 23:06:55.875833 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:55.875681 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" event={"ID":"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21","Type":"ContainerStarted","Data":"a93a9aa99860449fe0e288a81b70c10c80003d9ffe55e2027826670a6c92ee81"} Sep 09 23:06:55.889511 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:55.889467 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t69sp" event={"ID":"1c6b309f-307c-4a0f-a0d1-586fadcd13b1","Type":"ContainerStarted","Data":"aefc80c59c8df8029d382724a9e787f30e1b74bcad8d2c39858847b4622133f1"} Sep 09 23:06:55.895628 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:55.895585 2561 generic.go:356] "Generic (PLEG): container finished" podID="47e768c654fbb38de18e4644c0ab2928" containerID="a3ced9275ad08bd3509878f44755dddf3e02a1e7def5052741a93e3fdf0460a7" exitCode=0 Sep 09 23:06:55.895968 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:55.895921 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" event={"ID":"47e768c654fbb38de18e4644c0ab2928","Type":"ContainerDied","Data":"a3ced9275ad08bd3509878f44755dddf3e02a1e7def5052741a93e3fdf0460a7"} Sep 09 23:06:55.929277 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:55.929227 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-192.ec2.internal" event={"ID":"ef36ec20e9fe178e1d7d8c3ae65f1ea2","Type":"ContainerStarted","Data":"f1712a61cb974af94535979f11b6979bc87d5f712eceea3dec95e2bbbab3eba5"} Sep 09 23:06:55.939859 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:55.939815 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-25rz7" event={"ID":"5560171c-fecb-4328-a381-d6ecb4a7bcaf","Type":"ContainerStarted","Data":"6ca7212566bf6e0a2a4570806f7c79d66bd80f5627555c99047b745a19922bc8"} Sep 09 23:06:55.947951 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:55.947880 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d76sb" event={"ID":"d702c7a3-03a5-4b8b-9a7f-df843575cc43","Type":"ContainerStarted","Data":"d9dc06932ef23393bbfe595e13a4d1826e20268eb59b644310540c4a111ab69f"} Sep 09 23:06:56.966654 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:56.966615 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" event={"ID":"47e768c654fbb38de18e4644c0ab2928","Type":"ContainerStarted","Data":"959ad07f903249dd99a7ec4147b6b252970c005ffc9eb1572b7f0d0575719d44"} Sep 09 23:06:56.982950 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:56.982794 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-192.ec2.internal" podStartSLOduration=3.98274723 podStartE2EDuration="3.98274723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:06:55.943404389 +0000 UTC m=+4.629187798" watchObservedRunningTime="2026-09-09 23:06:56.98274723 +0000 UTC m=+5.668530641" Sep 09 23:06:56.983763 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:56.983314 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" podStartSLOduration=3.98329142 podStartE2EDuration="3.98329142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:06:56.981454534 +0000 UTC m=+5.667237941" watchObservedRunningTime="2026-09-09 23:06:56.98329142 +0000 UTC m=+5.669074828" Sep 09 23:06:57.420488 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:57.420392 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs\") pod \"network-metrics-daemon-j89rl\" (UID: \"9e224f1b-9ce8-484e-8de5-f4f4f49a5eab\") " pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:06:57.420488 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:57.420461 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31b8a991-a7b8-4cae-9125-69fcd001f700-original-pull-secret\") pod \"global-pull-secret-syncer-gw2dp\" (UID: \"31b8a991-a7b8-4cae-9125-69fcd001f700\") " pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:06:57.420713 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:57.420619 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:06:57.420713 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:57.420690 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31b8a991-a7b8-4cae-9125-69fcd001f700-original-pull-secret podName:31b8a991-a7b8-4cae-9125-69fcd001f700 nodeName:}" failed. No retries permitted until 2026-09-09 23:07:01.420671117 +0000 UTC m=+10.106454521 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31b8a991-a7b8-4cae-9125-69fcd001f700-original-pull-secret") pod "global-pull-secret-syncer-gw2dp" (UID: "31b8a991-a7b8-4cae-9125-69fcd001f700") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:06:57.421143 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:57.421121 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:06:57.421222 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:57.421185 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs podName:9e224f1b-9ce8-484e-8de5-f4f4f49a5eab nodeName:}" failed. No retries permitted until 2026-09-09 23:07:01.421168059 +0000 UTC m=+10.106951447 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs") pod "network-metrics-daemon-j89rl" (UID: "9e224f1b-9ce8-484e-8de5-f4f4f49a5eab") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:06:57.521757 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:57.521052 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t78d5\" (UniqueName: \"kubernetes.io/projected/a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd-kube-api-access-t78d5\") pod \"network-check-target-v7c8p\" (UID: \"a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd\") " pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:06:57.521757 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:57.521276 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:06:57.521757 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:57.521298 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:06:57.521757 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:57.521312 2561 projected.go:196] Error preparing data for projected volume kube-api-access-t78d5 for pod openshift-network-diagnostics/network-check-target-v7c8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:06:57.521757 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:57.521376 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd-kube-api-access-t78d5 podName:a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd nodeName:}" failed. No retries permitted until 2026-09-09 23:07:01.521356427 +0000 UTC m=+10.207139816 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t78d5" (UniqueName: "kubernetes.io/projected/a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd-kube-api-access-t78d5") pod "network-check-target-v7c8p" (UID: "a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:06:57.806519 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:57.805027 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:06:57.806519 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:57.805167 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gw2dp" podUID="31b8a991-a7b8-4cae-9125-69fcd001f700" Sep 09 23:06:57.806519 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:57.805632 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:06:57.806519 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:57.805732 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c8p" podUID="a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd" Sep 09 23:06:57.806519 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:57.805897 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:06:57.806519 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:57.805999 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j89rl" podUID="9e224f1b-9ce8-484e-8de5-f4f4f49a5eab" Sep 09 23:06:59.807235 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:59.806402 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:06:59.807235 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:59.806535 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c8p" podUID="a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd" Sep 09 23:06:59.807235 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:59.806946 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:06:59.807235 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:59.807052 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j89rl" podUID="9e224f1b-9ce8-484e-8de5-f4f4f49a5eab" Sep 09 23:06:59.807235 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:06:59.807112 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:06:59.807235 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:06:59.807189 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gw2dp" podUID="31b8a991-a7b8-4cae-9125-69fcd001f700" Sep 09 23:07:01.462568 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:01.462297 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs\") pod \"network-metrics-daemon-j89rl\" (UID: \"9e224f1b-9ce8-484e-8de5-f4f4f49a5eab\") " pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:07:01.463070 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:01.462601 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31b8a991-a7b8-4cae-9125-69fcd001f700-original-pull-secret\") pod \"global-pull-secret-syncer-gw2dp\" (UID: \"31b8a991-a7b8-4cae-9125-69fcd001f700\") " pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:07:01.463070 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:01.462498 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:07:01.463070 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:01.462726 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs podName:9e224f1b-9ce8-484e-8de5-f4f4f49a5eab nodeName:}" failed. No retries permitted until 2026-09-09 23:07:09.462689539 +0000 UTC m=+18.148472923 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs") pod "network-metrics-daemon-j89rl" (UID: "9e224f1b-9ce8-484e-8de5-f4f4f49a5eab") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:07:01.463070 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:01.462728 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:07:01.463070 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:01.462782 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31b8a991-a7b8-4cae-9125-69fcd001f700-original-pull-secret podName:31b8a991-a7b8-4cae-9125-69fcd001f700 nodeName:}" failed. No retries permitted until 2026-09-09 23:07:09.462768981 +0000 UTC m=+18.148552384 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31b8a991-a7b8-4cae-9125-69fcd001f700-original-pull-secret") pod "global-pull-secret-syncer-gw2dp" (UID: "31b8a991-a7b8-4cae-9125-69fcd001f700") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:07:01.563375 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:01.563281 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t78d5\" (UniqueName: \"kubernetes.io/projected/a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd-kube-api-access-t78d5\") pod \"network-check-target-v7c8p\" (UID: \"a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd\") " pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:07:01.563529 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:01.563448 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:07:01.563529 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:01.563474 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:07:01.563529 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:01.563488 2561 projected.go:196] Error preparing data for projected volume kube-api-access-t78d5 for pod openshift-network-diagnostics/network-check-target-v7c8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:07:01.563659 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:01.563553 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd-kube-api-access-t78d5 podName:a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd nodeName:}" failed. No retries permitted until 2026-09-09 23:07:09.563536559 +0000 UTC m=+18.249319945 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t78d5" (UniqueName: "kubernetes.io/projected/a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd-kube-api-access-t78d5") pod "network-check-target-v7c8p" (UID: "a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:07:01.805235 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:01.805118 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:07:01.805235 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:01.805222 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gw2dp" podUID="31b8a991-a7b8-4cae-9125-69fcd001f700" Sep 09 23:07:01.805604 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:01.805585 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:07:01.805700 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:01.805684 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c8p" podUID="a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd" Sep 09 23:07:01.805871 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:01.805836 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:07:01.805987 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:01.805966 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j89rl" podUID="9e224f1b-9ce8-484e-8de5-f4f4f49a5eab" Sep 09 23:07:03.803695 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:03.803607 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:07:03.804194 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:03.803752 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gw2dp" podUID="31b8a991-a7b8-4cae-9125-69fcd001f700" Sep 09 23:07:03.804194 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:03.804182 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:07:03.804321 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:03.804299 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c8p" podUID="a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd" Sep 09 23:07:03.804375 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:03.804331 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:07:03.804431 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:03.804385 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j89rl" podUID="9e224f1b-9ce8-484e-8de5-f4f4f49a5eab" Sep 09 23:07:05.804349 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:05.804317 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:07:05.804792 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:05.804314 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:07:05.804792 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:05.804463 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j89rl" podUID="9e224f1b-9ce8-484e-8de5-f4f4f49a5eab" Sep 09 23:07:05.804792 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:05.804313 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:07:05.804792 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:05.804516 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gw2dp" podUID="31b8a991-a7b8-4cae-9125-69fcd001f700" Sep 09 23:07:05.804792 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:05.804598 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c8p" podUID="a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd" Sep 09 23:07:07.803785 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:07.803748 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:07:07.803785 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:07.803768 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:07:07.804335 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:07.803748 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:07:07.804335 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:07.803907 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gw2dp" podUID="31b8a991-a7b8-4cae-9125-69fcd001f700" Sep 09 23:07:07.804335 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:07.804017 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c8p" podUID="a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd" Sep 09 23:07:07.804335 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:07.804094 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j89rl" podUID="9e224f1b-9ce8-484e-8de5-f4f4f49a5eab" Sep 09 23:07:09.526309 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:09.526225 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs\") pod \"network-metrics-daemon-j89rl\" (UID: \"9e224f1b-9ce8-484e-8de5-f4f4f49a5eab\") " pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:07:09.526309 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:09.526277 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31b8a991-a7b8-4cae-9125-69fcd001f700-original-pull-secret\") pod \"global-pull-secret-syncer-gw2dp\" (UID: \"31b8a991-a7b8-4cae-9125-69fcd001f700\") " pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:07:09.526760 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:09.526402 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:07:09.526760 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:09.526472 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs podName:9e224f1b-9ce8-484e-8de5-f4f4f49a5eab nodeName:}" failed. No retries permitted until 2026-09-09 23:07:25.526454845 +0000 UTC m=+34.212238226 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs") pod "network-metrics-daemon-j89rl" (UID: "9e224f1b-9ce8-484e-8de5-f4f4f49a5eab") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:07:09.526760 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:09.526407 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:07:09.526760 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:09.526515 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31b8a991-a7b8-4cae-9125-69fcd001f700-original-pull-secret podName:31b8a991-a7b8-4cae-9125-69fcd001f700 nodeName:}" failed. No retries permitted until 2026-09-09 23:07:25.526503035 +0000 UTC m=+34.212286417 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31b8a991-a7b8-4cae-9125-69fcd001f700-original-pull-secret") pod "global-pull-secret-syncer-gw2dp" (UID: "31b8a991-a7b8-4cae-9125-69fcd001f700") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:07:09.627096 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:09.627059 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t78d5\" (UniqueName: \"kubernetes.io/projected/a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd-kube-api-access-t78d5\") pod \"network-check-target-v7c8p\" (UID: \"a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd\") " pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:07:09.627303 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:09.627240 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:07:09.627303 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:09.627265 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:07:09.627303 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:09.627279 2561 projected.go:196] Error preparing data for projected volume kube-api-access-t78d5 for pod openshift-network-diagnostics/network-check-target-v7c8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:07:09.627448 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:09.627347 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd-kube-api-access-t78d5 podName:a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd nodeName:}" failed. No retries permitted until 2026-09-09 23:07:25.627328242 +0000 UTC m=+34.313111623 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t78d5" (UniqueName: "kubernetes.io/projected/a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd-kube-api-access-t78d5") pod "network-check-target-v7c8p" (UID: "a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:07:09.804392 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:09.804314 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:07:09.804561 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:09.804313 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:07:09.804561 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:09.804464 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c8p" podUID="a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd" Sep 09 23:07:09.804561 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:09.804324 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:07:09.804561 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:09.804513 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gw2dp" podUID="31b8a991-a7b8-4cae-9125-69fcd001f700" Sep 09 23:07:09.804758 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:09.804644 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j89rl" podUID="9e224f1b-9ce8-484e-8de5-f4f4f49a5eab" Sep 09 23:07:11.805044 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:11.805001 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:07:11.805529 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:11.805133 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gw2dp" podUID="31b8a991-a7b8-4cae-9125-69fcd001f700" Sep 09 23:07:11.805529 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:11.805146 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:07:11.805529 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:11.805167 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:07:11.805529 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:11.805237 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j89rl" podUID="9e224f1b-9ce8-484e-8de5-f4f4f49a5eab" Sep 09 23:07:11.805529 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:11.805298 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c8p" podUID="a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd" Sep 09 23:07:13.804197 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:13.803961 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:07:13.804770 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:13.803961 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:07:13.804770 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:13.804310 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gw2dp" podUID="31b8a991-a7b8-4cae-9125-69fcd001f700" Sep 09 23:07:13.804770 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:13.804349 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c8p" podUID="a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd" Sep 09 23:07:13.804770 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:13.804029 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:07:13.804770 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:13.804428 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j89rl" podUID="9e224f1b-9ce8-484e-8de5-f4f4f49a5eab" Sep 09 23:07:14.004779 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:14.004678 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d76sb" event={"ID":"d702c7a3-03a5-4b8b-9a7f-df843575cc43","Type":"ContainerStarted","Data":"c0def1d27eb76f3586971919a04000ac5c50b46cebe2e78669ec71cc033f43a4"} Sep 09 23:07:14.006654 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:14.006623 2561 generic.go:356] "Generic (PLEG): container finished" podID="b0c7b946-6758-4433-8081-5e3ea05b3639" containerID="39963ca88715c29816751ff8a1508c19425c8727026783aaa2e879710021e08b" exitCode=0 Sep 09 23:07:14.006819 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:14.006703 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zncr7" event={"ID":"b0c7b946-6758-4433-8081-5e3ea05b3639","Type":"ContainerDied","Data":"39963ca88715c29816751ff8a1508c19425c8727026783aaa2e879710021e08b"} Sep 09 23:07:14.013224 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:14.013188 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z9xqn" event={"ID":"d5402f8c-eefb-4fb3-9742-b96b8207198f","Type":"ContainerStarted","Data":"fda562e2b9e6e5d0fc3f0cc86d536007c7f15510e79256d6b3d4002b763f6458"} Sep 09 23:07:14.015537 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:14.015509 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" event={"ID":"08bcba00-9881-442c-aa02-b875c99817c2","Type":"ContainerStarted","Data":"4d5e5cb570947a015def91d1d3e44d77973bd33013bbaac862258e2673bf43a9"} Sep 09 23:07:14.017242 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:14.017220 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" event={"ID":"3881b43c-4c39-4800-a4fd-a060fb8cf5e8","Type":"ContainerStarted","Data":"9992c99a53ab97c3a26aa1c1edd720956267933e24b334303e752f144a6ee635"} Sep 09 23:07:14.018607 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:14.018585 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jqxbd" event={"ID":"a843f1d8-0cd1-4396-8fc1-5a22cd20f6e8","Type":"ContainerStarted","Data":"d0515b22c9d81f8c3fd7d99d3d6c19c0169537c663c783d7d68cb33074e474dd"} Sep 09 23:07:14.022147 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:14.022121 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" event={"ID":"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21","Type":"ContainerStarted","Data":"edb7842c1c24af5a987af0eb54d454aeb4fc521fa3c2da8e729ec7418feaa9cb"} Sep 09 23:07:14.022320 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:14.022200 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" event={"ID":"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21","Type":"ContainerStarted","Data":"041f406447ba27a9ebca1843fe537c9a5a3a9ccc15d91b9ce51d7aeb0c7dc7e3"} Sep 09 23:07:14.022320 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:14.022249 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" event={"ID":"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21","Type":"ContainerStarted","Data":"3a77ec20c61ac38ed9d260f725e2f692944b9152aae6f089a185c67dfe033aff"} Sep 09 23:07:14.022320 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:14.022295 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" event={"ID":"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21","Type":"ContainerStarted","Data":"33e5ef925f5e4ca24a674a5fad1b732b12ce60c440295db0b6e35626d1153d39"} Sep 09 23:07:14.022451 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:14.022341 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" event={"ID":"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21","Type":"ContainerStarted","Data":"ca6840cf94e5ba47d839f4e1ce223fd7fa0e9a85a5045125b5c5b13d32a91f7a"} Sep 09 23:07:14.022451 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:14.022390 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" event={"ID":"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21","Type":"ContainerStarted","Data":"90e9ff08778c40cfd92d926748b31aee20a2a71e97c1e4e47c93ad686e2307fb"} Sep 09 23:07:14.023624 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:14.023601 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t69sp" event={"ID":"1c6b309f-307c-4a0f-a0d1-586fadcd13b1","Type":"ContainerStarted","Data":"f98541a064506f29c8f52fa92e0fe70f25dc1c0113bd542fce8b63e285bd959f"} Sep 09 23:07:14.025386 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:14.025351 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-d76sb" podStartSLOduration=3.9539567030000002 podStartE2EDuration="22.025338402s" totalImagesPullingTime="18.071381699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:07:14.024893098 +0000 UTC m=+22.710676515" watchObservedRunningTime="2026-09-09 23:07:14.025338402 +0000 UTC m=+22.711121805" Sep 09 23:07:14.037067 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:14.036834 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-z9xqn" podStartSLOduration=7.887803861 podStartE2EDuration="22.036820158s" totalImagesPullingTime="14.149016297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:07:14.036575257 +0000 UTC m=+22.722358663" watchObservedRunningTime="2026-09-09 23:07:14.036820158 +0000 UTC m=+22.722603555" Sep 09 23:07:14.052619 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:14.052565 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jqxbd" podStartSLOduration=3.980290061 podStartE2EDuration="22.052548857s" totalImagesPullingTime="18.072258796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:07:14.052243479 +0000 UTC m=+22.738026893" watchObservedRunningTime="2026-09-09 23:07:14.052548857 +0000 UTC m=+22.738332260" Sep 09 23:07:14.084429 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:14.084380 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-t69sp" podStartSLOduration=3.947782177 podStartE2EDuration="22.084365836s" totalImagesPullingTime="18.136583659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:07:14.083962248 +0000 UTC m=+22.769745680" watchObservedRunningTime="2026-09-09 23:07:14.084365836 +0000 UTC m=+22.770149239" Sep 09 23:07:14.098100 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:14.098049 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5qpkc" podStartSLOduration=4.009274626 podStartE2EDuration="22.098030104s" totalImagesPullingTime="18.088755478s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:07:14.097884317 +0000 UTC m=+22.783667735" watchObservedRunningTime="2026-09-09 23:07:14.098030104 +0000 UTC m=+22.783813508" Sep 09 23:07:14.693333 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:14.693151 2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 09 23:07:14.741210 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:14.741097 2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T23:07:14.693327579Z","UUID":"10ae08db-4f77-461d-96cf-dd5fe63bf43f","Handler":null,"Name":"","Endpoint":""} Sep 09 23:07:14.742724 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:14.742696 2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 09 23:07:14.742870 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:14.742738 2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 09 23:07:15.027417 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:15.027383 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" event={"ID":"3881b43c-4c39-4800-a4fd-a060fb8cf5e8","Type":"ContainerStarted","Data":"e85a819436287fccb7122e68bd0081107b3e3a92d43abdab74581dee5ffe9121"} Sep 09 23:07:15.028798 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:15.028778 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-z9xqn" Sep 09 23:07:15.028953 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:15.028821 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-25rz7" event={"ID":"5560171c-fecb-4328-a381-d6ecb4a7bcaf","Type":"ContainerStarted","Data":"d57eec1a2746056d881a9c55096d2e7ebdcfec32ddb0f922475eb485e14954ff"} Sep 09 23:07:15.029487 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:15.029464 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-z9xqn" Sep 09 23:07:15.041929 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:15.041879 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-25rz7" podStartSLOduration=4.97340557 podStartE2EDuration="23.041846622s" totalImagesPullingTime="18.068441052s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:07:15.041501238 +0000 UTC m=+23.727284654" watchObservedRunningTime="2026-09-09 23:07:15.041846622 +0000 UTC m=+23.727630024" Sep 09 23:07:15.804016 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:15.803978 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:07:15.804181 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:15.803978 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:07:15.804181 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:15.804126 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gw2dp" podUID="31b8a991-a7b8-4cae-9125-69fcd001f700" Sep 09 23:07:15.804181 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:15.803978 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:07:15.804363 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:15.804178 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c8p" podUID="a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd" Sep 09 23:07:15.804363 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:15.804257 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j89rl" podUID="9e224f1b-9ce8-484e-8de5-f4f4f49a5eab" Sep 09 23:07:16.032621 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:16.032530 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" event={"ID":"3881b43c-4c39-4800-a4fd-a060fb8cf5e8","Type":"ContainerStarted","Data":"0d2f4a30670493354fc54be684725cfc96d562011af41a662bd82d84569c5b8a"} Sep 09 23:07:16.036083 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:16.036062 2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 09 23:07:16.036214 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:16.036063 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" event={"ID":"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21","Type":"ContainerStarted","Data":"f86ebf389d2b465eea0907cfca0fa799d57c0d7e516769e2f0a5812a7bdad54f"} Sep 09 23:07:16.047319 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:16.047275 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pft9k" podStartSLOduration=3.845527449 podStartE2EDuration="24.047259405s" totalImagesPullingTime="20.201731956s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:06:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:07:16.046885805 +0000 UTC m=+24.732669216" watchObservedRunningTime="2026-09-09 23:07:16.047259405 +0000 UTC m=+24.733042809" Sep 09 23:07:17.803763 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:17.803719 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:07:17.803763 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:17.803740 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:07:17.804236 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:17.803831 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:07:17.804236 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:17.803867 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gw2dp" podUID="31b8a991-a7b8-4cae-9125-69fcd001f700" Sep 09 23:07:17.804236 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:17.803930 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j89rl" podUID="9e224f1b-9ce8-484e-8de5-f4f4f49a5eab" Sep 09 23:07:17.804236 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:17.803994 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c8p" podUID="a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd" Sep 09 23:07:19.046547 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:19.046304 2561 generic.go:356] "Generic (PLEG): container finished" podID="b0c7b946-6758-4433-8081-5e3ea05b3639" containerID="8e721e515939078151c509d7d4ef3fb675b18feae0f954a231f397428b91ac52" exitCode=0 Sep 09 23:07:19.046547 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:19.046388 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zncr7" event={"ID":"b0c7b946-6758-4433-8081-5e3ea05b3639","Type":"ContainerDied","Data":"8e721e515939078151c509d7d4ef3fb675b18feae0f954a231f397428b91ac52"} Sep 09 23:07:19.050935 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:19.050904 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" event={"ID":"6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21","Type":"ContainerStarted","Data":"39c80decd1133012f00b51a52d2b986dbee179901e1290776c22271cc8147277"} Sep 09 23:07:19.051150 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:19.051136 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:07:19.067929 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:19.067899 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:07:19.086744 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:19.086699 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" podStartSLOduration=8.963239005 podStartE2EDuration="27.086683925s" totalImagesPullingTime="18.12344492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:07:19.086569214 +0000 UTC m=+27.772352617" watchObservedRunningTime="2026-09-09 23:07:19.086683925 +0000 UTC m=+27.772467325" Sep 09 23:07:19.810878 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:19.809608 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:07:19.810878 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:19.809743 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gw2dp" podUID="31b8a991-a7b8-4cae-9125-69fcd001f700" Sep 09 23:07:19.810878 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:19.810201 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:07:19.810878 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:19.810299 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c8p" podUID="a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd" Sep 09 23:07:19.810878 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:19.810457 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:07:19.810878 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:19.810558 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j89rl" podUID="9e224f1b-9ce8-484e-8de5-f4f4f49a5eab" Sep 09 23:07:20.054025 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:20.053998 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:07:20.054487 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:20.054070 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:07:20.069413 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:20.069329 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:07:20.448559 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:20.448341 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gw2dp"] Sep 09 23:07:20.448771 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:20.448753 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:07:20.448903 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:20.448881 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gw2dp" podUID="31b8a991-a7b8-4cae-9125-69fcd001f700" Sep 09 23:07:20.451366 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:20.451327 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v7c8p"] Sep 09 23:07:20.451509 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:20.451464 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:07:20.451578 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:20.451557 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c8p" podUID="a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd" Sep 09 23:07:20.452058 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:20.452037 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j89rl"] Sep 09 23:07:20.452168 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:20.452137 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:07:20.452244 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:20.452224 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j89rl" podUID="9e224f1b-9ce8-484e-8de5-f4f4f49a5eab" Sep 09 23:07:21.058541 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:21.058505 2561 generic.go:356] "Generic (PLEG): container finished" podID="b0c7b946-6758-4433-8081-5e3ea05b3639" containerID="434dbd4bf3f9fb15244e624bde52db02ac7d3f1066334e2aa53cf0eb7eb00640" exitCode=0 Sep 09 23:07:21.059191 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:21.058597 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zncr7" event={"ID":"b0c7b946-6758-4433-8081-5e3ea05b3639","Type":"ContainerDied","Data":"434dbd4bf3f9fb15244e624bde52db02ac7d3f1066334e2aa53cf0eb7eb00640"} Sep 09 23:07:21.326609 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:21.326481 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-z9xqn" Sep 09 23:07:21.326770 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:21.326628 2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 09 23:07:21.327070 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:21.327046 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-z9xqn" Sep 09 23:07:21.805097 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:21.805010 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:07:21.805253 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:21.805121 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gw2dp" podUID="31b8a991-a7b8-4cae-9125-69fcd001f700" Sep 09 23:07:21.805253 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:21.805167 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:07:21.805326 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:21.805264 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j89rl" podUID="9e224f1b-9ce8-484e-8de5-f4f4f49a5eab" Sep 09 23:07:22.803776 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:22.803745 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:07:22.804229 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:22.803889 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c8p" podUID="a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd" Sep 09 23:07:23.065790 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:23.065705 2561 generic.go:356] "Generic (PLEG): container finished" podID="b0c7b946-6758-4433-8081-5e3ea05b3639" containerID="5694c5a7f805dd46e576ee2f372cb4ad59cc0488697f764f4b213a13fb7cbec9" exitCode=0 Sep 09 23:07:23.065790 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:23.065763 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zncr7" event={"ID":"b0c7b946-6758-4433-8081-5e3ea05b3639","Type":"ContainerDied","Data":"5694c5a7f805dd46e576ee2f372cb4ad59cc0488697f764f4b213a13fb7cbec9"} Sep 09 23:07:23.803635 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:23.803603 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:07:23.803809 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:23.803743 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gw2dp" podUID="31b8a991-a7b8-4cae-9125-69fcd001f700" Sep 09 23:07:23.803809 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:23.803785 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:07:23.804189 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:23.803898 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j89rl" podUID="9e224f1b-9ce8-484e-8de5-f4f4f49a5eab" Sep 09 23:07:24.804151 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:24.804056 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:07:24.804611 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:24.804202 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7c8p" podUID="a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd" Sep 09 23:07:25.555501 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:25.555457 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs\") pod \"network-metrics-daemon-j89rl\" (UID: \"9e224f1b-9ce8-484e-8de5-f4f4f49a5eab\") " pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:07:25.555758 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:25.555521 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31b8a991-a7b8-4cae-9125-69fcd001f700-original-pull-secret\") pod \"global-pull-secret-syncer-gw2dp\" (UID: \"31b8a991-a7b8-4cae-9125-69fcd001f700\") " pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:07:25.555758 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:25.555646 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:07:25.555758 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:25.555674 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:07:25.555758 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:25.555729 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs podName:9e224f1b-9ce8-484e-8de5-f4f4f49a5eab nodeName:}" failed. No retries permitted until 2026-09-09 23:07:57.555707858 +0000 UTC m=+66.241491245 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs") pod "network-metrics-daemon-j89rl" (UID: "9e224f1b-9ce8-484e-8de5-f4f4f49a5eab") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:07:25.555758 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:25.555750 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31b8a991-a7b8-4cae-9125-69fcd001f700-original-pull-secret podName:31b8a991-a7b8-4cae-9125-69fcd001f700 nodeName:}" failed. No retries permitted until 2026-09-09 23:07:57.555740944 +0000 UTC m=+66.241524328 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31b8a991-a7b8-4cae-9125-69fcd001f700-original-pull-secret") pod "global-pull-secret-syncer-gw2dp" (UID: "31b8a991-a7b8-4cae-9125-69fcd001f700") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:07:25.656894 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:25.656839 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t78d5\" (UniqueName: \"kubernetes.io/projected/a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd-kube-api-access-t78d5\") pod \"network-check-target-v7c8p\" (UID: \"a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd\") " pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:07:25.657087 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:25.657017 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:07:25.657087 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:25.657045 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:07:25.657087 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:25.657059 2561 projected.go:196] Error preparing data for projected volume kube-api-access-t78d5 for pod openshift-network-diagnostics/network-check-target-v7c8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:07:25.657239 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:25.657136 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd-kube-api-access-t78d5 podName:a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd nodeName:}" failed. No retries permitted until 2026-09-09 23:07:57.65711432 +0000 UTC m=+66.342897708 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-t78d5" (UniqueName: "kubernetes.io/projected/a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd-kube-api-access-t78d5") pod "network-check-target-v7c8p" (UID: "a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:07:25.804175 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:25.804137 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:07:25.804612 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:25.804155 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:07:25.804612 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:25.804273 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gw2dp" podUID="31b8a991-a7b8-4cae-9125-69fcd001f700" Sep 09 23:07:25.804612 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:25.804325 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j89rl" podUID="9e224f1b-9ce8-484e-8de5-f4f4f49a5eab" Sep 09 23:07:26.099940 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.099907 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeReady" Sep 09 23:07:26.100125 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.100037 2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 09 23:07:26.143305 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.143210 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tdvkg"] Sep 09 23:07:26.148504 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.148473 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xbplg"] Sep 09 23:07:26.148647 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.148603 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tdvkg" Sep 09 23:07:26.150881 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.150841 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d4hxc\"" Sep 09 23:07:26.151022 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.150903 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 09 23:07:26.151022 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.150993 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 09 23:07:26.152661 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.152638 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xbplg" Sep 09 23:07:26.154214 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.154195 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xbplg"] Sep 09 23:07:26.154590 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.154571 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 09 23:07:26.154839 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.154821 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7cr8c\"" Sep 09 23:07:26.154961 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.154824 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 09 23:07:26.155105 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.155089 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 09 23:07:26.157410 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.157388 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tdvkg"] Sep 09 23:07:26.260760 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.260723 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls\") pod \"dns-default-tdvkg\" (UID: \"e9be36cc-dc4f-414f-98cb-ce27d9d70ff2\") " pod="openshift-dns/dns-default-tdvkg" Sep 09 23:07:26.260760 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.260765 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-tmp-dir\") pod \"dns-default-tdvkg\" (UID: \"e9be36cc-dc4f-414f-98cb-ce27d9d70ff2\") " pod="openshift-dns/dns-default-tdvkg" Sep 09 23:07:26.261055 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.260800 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8r9v\" (UniqueName: \"kubernetes.io/projected/e060ec1e-b4e7-4369-9387-b6817a2bcd4a-kube-api-access-h8r9v\") pod \"ingress-canary-xbplg\" (UID: \"e060ec1e-b4e7-4369-9387-b6817a2bcd4a\") " pod="openshift-ingress-canary/ingress-canary-xbplg" Sep 09 23:07:26.261055 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.260916 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e060ec1e-b4e7-4369-9387-b6817a2bcd4a-cert\") pod \"ingress-canary-xbplg\" (UID: \"e060ec1e-b4e7-4369-9387-b6817a2bcd4a\") " pod="openshift-ingress-canary/ingress-canary-xbplg" Sep 09 23:07:26.261055 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.260939 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7hf9\" (UniqueName: \"kubernetes.io/projected/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-kube-api-access-f7hf9\") pod \"dns-default-tdvkg\" (UID: \"e9be36cc-dc4f-414f-98cb-ce27d9d70ff2\") " pod="openshift-dns/dns-default-tdvkg" Sep 09 23:07:26.261055 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.260975 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-config-volume\") pod \"dns-default-tdvkg\" (UID: \"e9be36cc-dc4f-414f-98cb-ce27d9d70ff2\") " pod="openshift-dns/dns-default-tdvkg" Sep 09 23:07:26.362104 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.362071 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls\") pod \"dns-default-tdvkg\" (UID: \"e9be36cc-dc4f-414f-98cb-ce27d9d70ff2\") " pod="openshift-dns/dns-default-tdvkg" Sep 09 23:07:26.362104 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.362106 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-tmp-dir\") pod \"dns-default-tdvkg\" (UID: \"e9be36cc-dc4f-414f-98cb-ce27d9d70ff2\") " pod="openshift-dns/dns-default-tdvkg" Sep 09 23:07:26.362355 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.362134 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8r9v\" (UniqueName: \"kubernetes.io/projected/e060ec1e-b4e7-4369-9387-b6817a2bcd4a-kube-api-access-h8r9v\") pod \"ingress-canary-xbplg\" (UID: \"e060ec1e-b4e7-4369-9387-b6817a2bcd4a\") " pod="openshift-ingress-canary/ingress-canary-xbplg" Sep 09 23:07:26.362355 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.362219 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e060ec1e-b4e7-4369-9387-b6817a2bcd4a-cert\") pod \"ingress-canary-xbplg\" (UID: \"e060ec1e-b4e7-4369-9387-b6817a2bcd4a\") " pod="openshift-ingress-canary/ingress-canary-xbplg" Sep 09 23:07:26.362355 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:26.362256 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:07:26.362355 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.362260 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7hf9\" (UniqueName: \"kubernetes.io/projected/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-kube-api-access-f7hf9\") pod \"dns-default-tdvkg\" (UID: \"e9be36cc-dc4f-414f-98cb-ce27d9d70ff2\") " pod="openshift-dns/dns-default-tdvkg" Sep 09 23:07:26.362355 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:26.362335 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls podName:e9be36cc-dc4f-414f-98cb-ce27d9d70ff2 nodeName:}" failed. No retries permitted until 2026-09-09 23:07:26.862311984 +0000 UTC m=+35.548095365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls") pod "dns-default-tdvkg" (UID: "e9be36cc-dc4f-414f-98cb-ce27d9d70ff2") : secret "dns-default-metrics-tls" not found Sep 09 23:07:26.362580 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.362392 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-config-volume\") pod \"dns-default-tdvkg\" (UID: \"e9be36cc-dc4f-414f-98cb-ce27d9d70ff2\") " pod="openshift-dns/dns-default-tdvkg" Sep 09 23:07:26.362767 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.362751 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-tmp-dir\") pod \"dns-default-tdvkg\" (UID: \"e9be36cc-dc4f-414f-98cb-ce27d9d70ff2\") " pod="openshift-dns/dns-default-tdvkg" Sep 09 23:07:26.363007 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.362985 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-config-volume\") pod \"dns-default-tdvkg\" (UID: \"e9be36cc-dc4f-414f-98cb-ce27d9d70ff2\") " pod="openshift-dns/dns-default-tdvkg" Sep 09 23:07:26.367548 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.367521 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e060ec1e-b4e7-4369-9387-b6817a2bcd4a-cert\") pod \"ingress-canary-xbplg\" (UID: \"e060ec1e-b4e7-4369-9387-b6817a2bcd4a\") " pod="openshift-ingress-canary/ingress-canary-xbplg" Sep 09 23:07:26.370833 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.370781 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7hf9\" (UniqueName: \"kubernetes.io/projected/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-kube-api-access-f7hf9\") pod \"dns-default-tdvkg\" (UID: \"e9be36cc-dc4f-414f-98cb-ce27d9d70ff2\") " pod="openshift-dns/dns-default-tdvkg" Sep 09 23:07:26.370971 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.370879 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8r9v\" (UniqueName: \"kubernetes.io/projected/e060ec1e-b4e7-4369-9387-b6817a2bcd4a-kube-api-access-h8r9v\") pod \"ingress-canary-xbplg\" (UID: \"e060ec1e-b4e7-4369-9387-b6817a2bcd4a\") " pod="openshift-ingress-canary/ingress-canary-xbplg" Sep 09 23:07:26.469810 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.469713 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xbplg" Sep 09 23:07:26.804377 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.804291 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:07:26.806720 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.806695 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t7jc8\"" Sep 09 23:07:26.806873 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.806785 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 23:07:26.806945 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.806695 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 23:07:26.867282 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:26.867249 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls\") pod \"dns-default-tdvkg\" (UID: \"e9be36cc-dc4f-414f-98cb-ce27d9d70ff2\") " pod="openshift-dns/dns-default-tdvkg" Sep 09 23:07:26.867473 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:26.867415 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:07:26.867535 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:26.867508 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls podName:e9be36cc-dc4f-414f-98cb-ce27d9d70ff2 nodeName:}" failed. No retries permitted until 2026-09-09 23:07:27.867484211 +0000 UTC m=+36.553267611 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls") pod "dns-default-tdvkg" (UID: "e9be36cc-dc4f-414f-98cb-ce27d9d70ff2") : secret "dns-default-metrics-tls" not found Sep 09 23:07:27.804202 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:27.804010 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:07:27.804354 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:27.804081 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:07:27.806687 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:27.806651 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h952m\"" Sep 09 23:07:27.806687 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:27.806671 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 23:07:27.807278 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:27.806718 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 23:07:27.851006 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:27.850979 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xbplg"] Sep 09 23:07:27.875454 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:27.875427 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls\") pod \"dns-default-tdvkg\" (UID: \"e9be36cc-dc4f-414f-98cb-ce27d9d70ff2\") " pod="openshift-dns/dns-default-tdvkg" Sep 09 23:07:27.875581 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:27.875565 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:07:27.875631 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:27.875622 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls podName:e9be36cc-dc4f-414f-98cb-ce27d9d70ff2 nodeName:}" failed. No retries permitted until 2026-09-09 23:07:29.875608268 +0000 UTC m=+38.561391649 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls") pod "dns-default-tdvkg" (UID: "e9be36cc-dc4f-414f-98cb-ce27d9d70ff2") : secret "dns-default-metrics-tls" not found Sep 09 23:07:27.894815 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:07:27.894788 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode060ec1e_b4e7_4369_9387_b6817a2bcd4a.slice/crio-b019a2a7240d8a923ff6be644f983d731044497a710f3adba27d2f47ba26e90d WatchSource:0}: Error finding container b019a2a7240d8a923ff6be644f983d731044497a710f3adba27d2f47ba26e90d: Status 404 returned error can't find the container with id b019a2a7240d8a923ff6be644f983d731044497a710f3adba27d2f47ba26e90d Sep 09 23:07:28.078163 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:28.078130 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xbplg" event={"ID":"e060ec1e-b4e7-4369-9387-b6817a2bcd4a","Type":"ContainerStarted","Data":"b019a2a7240d8a923ff6be644f983d731044497a710f3adba27d2f47ba26e90d"} Sep 09 23:07:28.080891 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:28.080864 2561 generic.go:356] "Generic (PLEG): container finished" podID="b0c7b946-6758-4433-8081-5e3ea05b3639" containerID="de22394e44112d227a67fceee22e50dc6dff52487734f8deede418ad686913aa" exitCode=0 Sep 09 23:07:28.081004 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:28.080904 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zncr7" event={"ID":"b0c7b946-6758-4433-8081-5e3ea05b3639","Type":"ContainerDied","Data":"de22394e44112d227a67fceee22e50dc6dff52487734f8deede418ad686913aa"} Sep 09 23:07:29.090656 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:29.090615 2561 generic.go:356] "Generic (PLEG): container finished" podID="b0c7b946-6758-4433-8081-5e3ea05b3639" containerID="2c97fc35783096f869d1d05ee6bd0e41c46407359416f78bedbcbcceee8d2087" exitCode=0 Sep 09 23:07:29.091359 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:29.090701 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zncr7" event={"ID":"b0c7b946-6758-4433-8081-5e3ea05b3639","Type":"ContainerDied","Data":"2c97fc35783096f869d1d05ee6bd0e41c46407359416f78bedbcbcceee8d2087"} Sep 09 23:07:29.891648 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:29.891603 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls\") pod \"dns-default-tdvkg\" (UID: \"e9be36cc-dc4f-414f-98cb-ce27d9d70ff2\") " pod="openshift-dns/dns-default-tdvkg" Sep 09 23:07:29.891821 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:29.891790 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:07:29.891923 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:29.891887 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls podName:e9be36cc-dc4f-414f-98cb-ce27d9d70ff2 nodeName:}" failed. No retries permitted until 2026-09-09 23:07:33.891842695 +0000 UTC m=+42.577626079 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls") pod "dns-default-tdvkg" (UID: "e9be36cc-dc4f-414f-98cb-ce27d9d70ff2") : secret "dns-default-metrics-tls" not found Sep 09 23:07:30.098261 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:30.098218 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zncr7" event={"ID":"b0c7b946-6758-4433-8081-5e3ea05b3639","Type":"ContainerStarted","Data":"d4ecc4463bcc6324d9534d1af7cfa33f223de4788e9a38ace95d62e848251301"} Sep 09 23:07:30.119136 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:30.119076 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zncr7" podStartSLOduration=8.412795838 podStartE2EDuration="38.119057812s" totalImagesPullingTime="29.706261974s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:06:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:07:30.11723764 +0000 UTC m=+38.803021059" watchObservedRunningTime="2026-09-09 23:07:30.119057812 +0000 UTC m=+38.804841228" Sep 09 23:07:31.102362 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:31.102326 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xbplg" event={"ID":"e060ec1e-b4e7-4369-9387-b6817a2bcd4a","Type":"ContainerStarted","Data":"cd44015ceaf00a5e66c8028c55dfe68f09eebce7ab026978f163bce33703f4e2"} Sep 09 23:07:31.118367 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:31.118314 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xbplg" podStartSLOduration=2.542973344 podStartE2EDuration="5.118299827s" totalImagesPullingTime="2.575326483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:07:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:07:31.117576537 +0000 UTC m=+39.803359952" watchObservedRunningTime="2026-09-09 23:07:31.118299827 +0000 UTC m=+39.804083227" Sep 09 23:07:33.917163 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:33.917121 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls\") pod \"dns-default-tdvkg\" (UID: \"e9be36cc-dc4f-414f-98cb-ce27d9d70ff2\") " pod="openshift-dns/dns-default-tdvkg" Sep 09 23:07:33.917560 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:33.917286 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:07:33.917560 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:33.917354 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls podName:e9be36cc-dc4f-414f-98cb-ce27d9d70ff2 nodeName:}" failed. No retries permitted until 2026-09-09 23:07:41.917337037 +0000 UTC m=+50.603120418 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls") pod "dns-default-tdvkg" (UID: "e9be36cc-dc4f-414f-98cb-ce27d9d70ff2") : secret "dns-default-metrics-tls" not found Sep 09 23:07:41.975524 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:41.975477 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls\") pod \"dns-default-tdvkg\" (UID: \"e9be36cc-dc4f-414f-98cb-ce27d9d70ff2\") " pod="openshift-dns/dns-default-tdvkg" Sep 09 23:07:41.975986 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:41.975597 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:07:41.975986 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:41.975660 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls podName:e9be36cc-dc4f-414f-98cb-ce27d9d70ff2 nodeName:}" failed. No retries permitted until 2026-09-09 23:07:57.975644664 +0000 UTC m=+66.661428045 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls") pod "dns-default-tdvkg" (UID: "e9be36cc-dc4f-414f-98cb-ce27d9d70ff2") : secret "dns-default-metrics-tls" not found Sep 09 23:07:52.071725 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:52.071693 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pbrgn" Sep 09 23:07:57.582307 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:57.582266 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs\") pod \"network-metrics-daemon-j89rl\" (UID: \"9e224f1b-9ce8-484e-8de5-f4f4f49a5eab\") " pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:07:57.582307 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:57.582312 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31b8a991-a7b8-4cae-9125-69fcd001f700-original-pull-secret\") pod \"global-pull-secret-syncer-gw2dp\" (UID: \"31b8a991-a7b8-4cae-9125-69fcd001f700\") " pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:07:57.584800 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:57.584779 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 23:07:57.584885 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:57.584840 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 23:07:57.592798 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:57.592776 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 23:07:57.592883 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:57.592872 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs podName:9e224f1b-9ce8-484e-8de5-f4f4f49a5eab nodeName:}" failed. No retries permitted until 2026-09-09 23:09:01.592837527 +0000 UTC m=+130.278620911 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs") pod "network-metrics-daemon-j89rl" (UID: "9e224f1b-9ce8-484e-8de5-f4f4f49a5eab") : secret "metrics-daemon-secret" not found Sep 09 23:07:57.595885 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:57.595834 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31b8a991-a7b8-4cae-9125-69fcd001f700-original-pull-secret\") pod \"global-pull-secret-syncer-gw2dp\" (UID: \"31b8a991-a7b8-4cae-9125-69fcd001f700\") " pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:07:57.682756 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:57.682710 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t78d5\" (UniqueName: \"kubernetes.io/projected/a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd-kube-api-access-t78d5\") pod \"network-check-target-v7c8p\" (UID: \"a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd\") " pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:07:57.685320 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:57.685301 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 23:07:57.695187 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:57.695169 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 23:07:57.706536 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:57.706510 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t78d5\" (UniqueName: \"kubernetes.io/projected/a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd-kube-api-access-t78d5\") pod \"network-check-target-v7c8p\" (UID: \"a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd\") " pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:07:57.716633 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:57.716605 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t7jc8\"" Sep 09 23:07:57.724573 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:57.724540 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:07:57.820126 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:57.820053 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gw2dp" Sep 09 23:07:57.984832 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:57.984796 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls\") pod \"dns-default-tdvkg\" (UID: \"e9be36cc-dc4f-414f-98cb-ce27d9d70ff2\") " pod="openshift-dns/dns-default-tdvkg" Sep 09 23:07:57.985056 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:57.985032 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:07:57.985119 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:07:57.985109 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls podName:e9be36cc-dc4f-414f-98cb-ce27d9d70ff2 nodeName:}" failed. No retries permitted until 2026-09-09 23:08:29.985091727 +0000 UTC m=+98.670875108 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls") pod "dns-default-tdvkg" (UID: "e9be36cc-dc4f-414f-98cb-ce27d9d70ff2") : secret "dns-default-metrics-tls" not found Sep 09 23:07:58.019467 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:58.019431 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v7c8p"] Sep 09 23:07:58.022942 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:07:58.022919 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b97ef2_f3f6_4da7_8b42_e99128dd6ffd.slice/crio-6204c5cdb246ba42d8435c3ff4d6bb4724c4a18b3ce8ebb352f7354c84ea9a23 WatchSource:0}: Error finding container 6204c5cdb246ba42d8435c3ff4d6bb4724c4a18b3ce8ebb352f7354c84ea9a23: Status 404 returned error can't find the container with id 6204c5cdb246ba42d8435c3ff4d6bb4724c4a18b3ce8ebb352f7354c84ea9a23 Sep 09 23:07:58.096769 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:58.096694 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gw2dp"] Sep 09 23:07:58.099752 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:07:58.099726 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b8a991_a7b8_4cae_9125_69fcd001f700.slice/crio-ec7646359ee23c0751672deb99ae84acdd7103deecf15cb3562e382e3c07c486 WatchSource:0}: Error finding container ec7646359ee23c0751672deb99ae84acdd7103deecf15cb3562e382e3c07c486: Status 404 returned error can't find the container with id ec7646359ee23c0751672deb99ae84acdd7103deecf15cb3562e382e3c07c486 Sep 09 23:07:58.169259 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:58.169219 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gw2dp" event={"ID":"31b8a991-a7b8-4cae-9125-69fcd001f700","Type":"ContainerStarted","Data":"ec7646359ee23c0751672deb99ae84acdd7103deecf15cb3562e382e3c07c486"} Sep 09 23:07:58.170403 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:07:58.170373 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v7c8p" event={"ID":"a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd","Type":"ContainerStarted","Data":"6204c5cdb246ba42d8435c3ff4d6bb4724c4a18b3ce8ebb352f7354c84ea9a23"} Sep 09 23:08:03.186562 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:03.186513 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gw2dp" event={"ID":"31b8a991-a7b8-4cae-9125-69fcd001f700","Type":"ContainerStarted","Data":"b3698b479ae2c6fbc4ec066cc7a6136d2d2b69ae6e63874b0774a2a2dedde23f"} Sep 09 23:08:03.188257 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:03.188228 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v7c8p" event={"ID":"a0b97ef2-f3f6-4da7-8b42-e99128dd6ffd","Type":"ContainerStarted","Data":"5c4f3dc2a333710b7e9389564473fb21a3b4cd8b990e6f34a4f649692dfde444"} Sep 09 23:08:03.188400 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:03.188381 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:08:03.200303 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:03.200234 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gw2dp" podStartSLOduration=66.26245831 podStartE2EDuration="1m11.200217381s" totalImagesPullingTime="4.937759071s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:08:03.200108053 +0000 UTC m=+71.885891482" watchObservedRunningTime="2026-09-09 23:08:03.200217381 +0000 UTC m=+71.886000778" Sep 09 23:08:03.212952 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:03.212896 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-v7c8p" podStartSLOduration=66.198348022 podStartE2EDuration="1m11.212877095s" totalImagesPullingTime="5.014529073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:08:03.212619201 +0000 UTC m=+71.898402603" watchObservedRunningTime="2026-09-09 23:08:03.212877095 +0000 UTC m=+71.898660523" Sep 09 23:08:30.007381 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:30.007350 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls\") pod \"dns-default-tdvkg\" (UID: \"e9be36cc-dc4f-414f-98cb-ce27d9d70ff2\") " pod="openshift-dns/dns-default-tdvkg" Sep 09 23:08:30.007718 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:08:30.007508 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:08:30.007718 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:08:30.007585 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls podName:e9be36cc-dc4f-414f-98cb-ce27d9d70ff2 nodeName:}" failed. No retries permitted until 2026-09-09 23:09:34.007568143 +0000 UTC m=+162.693351523 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls") pod "dns-default-tdvkg" (UID: "e9be36cc-dc4f-414f-98cb-ce27d9d70ff2") : secret "dns-default-metrics-tls" not found Sep 09 23:08:34.194253 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:34.194216 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-v7c8p" Sep 09 23:08:56.413986 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:56.413949 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579b867994-qz5gz"] Sep 09 23:08:56.415595 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:56.415578 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579b867994-qz5gz" Sep 09 23:08:56.418971 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:56.418948 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-bmxq9\"" Sep 09 23:08:56.419110 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:56.418946 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 09 23:08:56.419110 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:56.418982 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 09 23:08:56.419231 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:56.419144 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 09 23:08:56.419231 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:56.419216 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 09 23:08:56.426919 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:56.426897 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579b867994-qz5gz"] Sep 09 23:08:56.492583 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:56.492541 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bczsk\" (UniqueName: \"kubernetes.io/projected/02ac530c-6761-4265-b864-6af6f318b272-kube-api-access-bczsk\") pod \"managed-serviceaccount-addon-agent-579b867994-qz5gz\" (UID: \"02ac530c-6761-4265-b864-6af6f318b272\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579b867994-qz5gz" Sep 09 23:08:56.492758 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:56.492592 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/02ac530c-6761-4265-b864-6af6f318b272-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-579b867994-qz5gz\" (UID: \"02ac530c-6761-4265-b864-6af6f318b272\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579b867994-qz5gz" Sep 09 23:08:56.593605 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:56.593560 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bczsk\" (UniqueName: \"kubernetes.io/projected/02ac530c-6761-4265-b864-6af6f318b272-kube-api-access-bczsk\") pod \"managed-serviceaccount-addon-agent-579b867994-qz5gz\" (UID: \"02ac530c-6761-4265-b864-6af6f318b272\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579b867994-qz5gz" Sep 09 23:08:56.593788 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:56.593619 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/02ac530c-6761-4265-b864-6af6f318b272-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-579b867994-qz5gz\" (UID: \"02ac530c-6761-4265-b864-6af6f318b272\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579b867994-qz5gz" Sep 09 23:08:56.596392 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:56.596361 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/02ac530c-6761-4265-b864-6af6f318b272-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-579b867994-qz5gz\" (UID: \"02ac530c-6761-4265-b864-6af6f318b272\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579b867994-qz5gz" Sep 09 23:08:56.602304 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:56.602277 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bczsk\" (UniqueName: \"kubernetes.io/projected/02ac530c-6761-4265-b864-6af6f318b272-kube-api-access-bczsk\") pod \"managed-serviceaccount-addon-agent-579b867994-qz5gz\" (UID: \"02ac530c-6761-4265-b864-6af6f318b272\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579b867994-qz5gz" Sep 09 23:08:56.737770 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:56.737724 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579b867994-qz5gz" Sep 09 23:08:57.006079 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:57.005992 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579b867994-qz5gz"] Sep 09 23:08:57.009135 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:08:57.009102 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02ac530c_6761_4265_b864_6af6f318b272.slice/crio-a45f739e0238bcbba01a8886b06e44d160dcae88bf33290bace14c0e4a595dce WatchSource:0}: Error finding container a45f739e0238bcbba01a8886b06e44d160dcae88bf33290bace14c0e4a595dce: Status 404 returned error can't find the container with id a45f739e0238bcbba01a8886b06e44d160dcae88bf33290bace14c0e4a595dce Sep 09 23:08:57.323669 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:57.323577 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579b867994-qz5gz" event={"ID":"02ac530c-6761-4265-b864-6af6f318b272","Type":"ContainerStarted","Data":"a45f739e0238bcbba01a8886b06e44d160dcae88bf33290bace14c0e4a595dce"} Sep 09 23:08:59.330676 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:59.330643 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579b867994-qz5gz" event={"ID":"02ac530c-6761-4265-b864-6af6f318b272","Type":"ContainerStarted","Data":"a9ce785ed81808596ecad0e0006a31ddfcf43ab27820731c31acc43d37c30e86"} Sep 09 23:08:59.343512 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:08:59.343460 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579b867994-qz5gz" podStartSLOduration=1.165963989 podStartE2EDuration="3.343442197s" totalImagesPullingTime="2.177478208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:08:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:08:59.342733499 +0000 UTC m=+128.028516901" watchObservedRunningTime="2026-09-09 23:08:59.343442197 +0000 UTC m=+128.029225601" Sep 09 23:09:01.631412 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:09:01.631349 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs\") pod \"network-metrics-daemon-j89rl\" (UID: \"9e224f1b-9ce8-484e-8de5-f4f4f49a5eab\") " pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:09:01.631843 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:09:01.631515 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 23:09:01.631843 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:09:01.631583 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs podName:9e224f1b-9ce8-484e-8de5-f4f4f49a5eab nodeName:}" failed. No retries permitted until 2026-09-09 23:11:03.6315663 +0000 UTC m=+252.317349681 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs") pod "network-metrics-daemon-j89rl" (UID: "9e224f1b-9ce8-484e-8de5-f4f4f49a5eab") : secret "metrics-daemon-secret" not found Sep 09 23:09:29.162620 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:09:29.162562 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-tdvkg" podUID="e9be36cc-dc4f-414f-98cb-ce27d9d70ff2" Sep 09 23:09:29.405914 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:09:29.405881 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tdvkg" Sep 09 23:09:30.815026 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:09:30.814979 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-j89rl" podUID="9e224f1b-9ce8-484e-8de5-f4f4f49a5eab" Sep 09 23:09:34.065808 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:09:34.065747 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls\") pod \"dns-default-tdvkg\" (UID: \"e9be36cc-dc4f-414f-98cb-ce27d9d70ff2\") " pod="openshift-dns/dns-default-tdvkg" Sep 09 23:09:34.066268 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:09:34.065928 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:09:34.066268 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:09:34.066013 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls podName:e9be36cc-dc4f-414f-98cb-ce27d9d70ff2 nodeName:}" failed. No retries permitted until 2026-09-09 23:11:36.065997722 +0000 UTC m=+284.751781103 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls") pod "dns-default-tdvkg" (UID: "e9be36cc-dc4f-414f-98cb-ce27d9d70ff2") : secret "dns-default-metrics-tls" not found Sep 09 23:09:41.805231 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:09:41.805164 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:10:45.422529 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.422491 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-b46465886-mcs6c"] Sep 09 23:10:45.425362 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.425343 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:10:45.431634 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.431611 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-n9ws5\"" Sep 09 23:10:45.433424 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.433409 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 09 23:10:45.434290 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.434270 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 09 23:10:45.434370 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.434279 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 09 23:10:45.434431 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.434279 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 09 23:10:45.434674 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.434658 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 09 23:10:45.435032 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.435009 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 09 23:10:45.461622 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.461592 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b46465886-mcs6c"] Sep 09 23:10:45.507289 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.507255 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-8jk5k"] Sep 09 23:10:45.510065 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.510044 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jk5k" Sep 09 23:10:45.512314 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.512289 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 09 23:10:45.512443 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.512290 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 09 23:10:45.512443 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.512339 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-27zhf\"" Sep 09 23:10:45.512443 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.512357 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 09 23:10:45.512719 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.512704 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 09 23:10:45.514618 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.513998 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rzk8q"] Sep 09 23:10:45.517003 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.516804 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-2fs7f"] Sep 09 23:10:45.517003 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.516996 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rzk8q" Sep 09 23:10:45.519461 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.519441 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56cd69fd8c-bj9b8"] Sep 09 23:10:45.519552 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.519488 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 09 23:10:45.519552 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.519532 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-8bxg6\"" Sep 09 23:10:45.519654 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.519579 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-2fs7f" Sep 09 23:10:45.520139 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.520119 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 09 23:10:45.522113 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.522099 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:45.522360 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.522342 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 09 23:10:45.522794 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.522777 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 09 23:10:45.523483 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.523460 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 09 23:10:45.523573 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.523491 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-zpdxg\"" Sep 09 23:10:45.523763 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.523744 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 09 23:10:45.524201 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.524172 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-8jk5k"] Sep 09 23:10:45.524295 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.524208 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 09 23:10:45.524609 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.524590 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 09 23:10:45.524746 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.524617 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 09 23:10:45.525301 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.525282 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p9f5w\"" Sep 09 23:10:45.531107 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.531067 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 09 23:10:45.531298 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.531226 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-2fs7f"] Sep 09 23:10:45.532161 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.532143 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 09 23:10:45.532349 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.532331 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rzk8q"] Sep 09 23:10:45.541967 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.541942 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56cd69fd8c-bj9b8"] Sep 09 23:10:45.572026 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.571986 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fa8702e-a63e-43aa-907c-e411e4af1b11-service-ca-bundle\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:10:45.572026 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.572033 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6fa8702e-a63e-43aa-907c-e411e4af1b11-stats-auth\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:10:45.572274 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.572068 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq2v9\" (UniqueName: \"kubernetes.io/projected/6fa8702e-a63e-43aa-907c-e411e4af1b11-kube-api-access-jq2v9\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:10:45.572274 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.572141 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa8702e-a63e-43aa-907c-e411e4af1b11-metrics-certs\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:10:45.572274 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.572173 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6fa8702e-a63e-43aa-907c-e411e4af1b11-default-certificate\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:10:45.672885 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.672758 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa8702e-a63e-43aa-907c-e411e4af1b11-metrics-certs\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:10:45.672885 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.672803 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/122b24bb-be61-4aa7-8faf-38f4c1f395f3-config\") pod \"service-ca-operator-859f94c4b-8jk5k\" (UID: \"122b24bb-be61-4aa7-8faf-38f4c1f395f3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jk5k" Sep 09 23:10:45.672885 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.672821 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02b665a3-2525-4102-87eb-419606cea310-trusted-ca\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:45.672885 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.672839 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02b665a3-2525-4102-87eb-419606cea310-ca-trust-extracted\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:45.672885 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.672894 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq2v9\" (UniqueName: \"kubernetes.io/projected/6fa8702e-a63e-43aa-907c-e411e4af1b11-kube-api-access-jq2v9\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:10:45.673287 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.672915 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fa8702e-a63e-43aa-907c-e411e4af1b11-service-ca-bundle\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:10:45.673287 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.672929 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6fa8702e-a63e-43aa-907c-e411e4af1b11-stats-auth\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:10:45.673287 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:45.672935 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 09 23:10:45.673287 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:45.673012 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fa8702e-a63e-43aa-907c-e411e4af1b11-metrics-certs podName:6fa8702e-a63e-43aa-907c-e411e4af1b11 nodeName:}" failed. No retries permitted until 2026-09-09 23:10:46.172995315 +0000 UTC m=+234.858778697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fa8702e-a63e-43aa-907c-e411e4af1b11-metrics-certs") pod "router-default-b46465886-mcs6c" (UID: "6fa8702e-a63e-43aa-907c-e411e4af1b11") : secret "router-metrics-certs-default" not found Sep 09 23:10:45.673287 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.672943 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a127828-4a97-40cf-b8ae-f5ecd20f97c6-serving-cert\") pod \"console-operator-85b454874b-2fs7f\" (UID: \"0a127828-4a97-40cf-b8ae-f5ecd20f97c6\") " pod="openshift-console-operator/console-operator-85b454874b-2fs7f" Sep 09 23:10:45.673287 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:45.673072 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6fa8702e-a63e-43aa-907c-e411e4af1b11-service-ca-bundle podName:6fa8702e-a63e-43aa-907c-e411e4af1b11 nodeName:}" failed. No retries permitted until 2026-09-09 23:10:46.173052979 +0000 UTC m=+234.858836375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6fa8702e-a63e-43aa-907c-e411e4af1b11-service-ca-bundle") pod "router-default-b46465886-mcs6c" (UID: "6fa8702e-a63e-43aa-907c-e411e4af1b11") : configmap references non-existent config key: service-ca.crt Sep 09 23:10:45.673287 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.673117 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjbl2\" (UniqueName: \"kubernetes.io/projected/0a127828-4a97-40cf-b8ae-f5ecd20f97c6-kube-api-access-sjbl2\") pod \"console-operator-85b454874b-2fs7f\" (UID: \"0a127828-4a97-40cf-b8ae-f5ecd20f97c6\") " pod="openshift-console-operator/console-operator-85b454874b-2fs7f" Sep 09 23:10:45.673287 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.673153 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02b665a3-2525-4102-87eb-419606cea310-registry-certificates\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:45.673287 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.673189 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkgn9\" (UniqueName: \"kubernetes.io/projected/72f66107-ac74-49f5-9782-583434006631-kube-api-access-qkgn9\") pod \"volume-data-source-validator-695c895bd6-rzk8q\" (UID: \"72f66107-ac74-49f5-9782-583434006631\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rzk8q" Sep 09 23:10:45.673287 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.673235 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-registry-tls\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:45.673287 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.673264 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6fa8702e-a63e-43aa-907c-e411e4af1b11-default-certificate\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:10:45.673772 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.673311 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a127828-4a97-40cf-b8ae-f5ecd20f97c6-trusted-ca\") pod \"console-operator-85b454874b-2fs7f\" (UID: \"0a127828-4a97-40cf-b8ae-f5ecd20f97c6\") " pod="openshift-console-operator/console-operator-85b454874b-2fs7f" Sep 09 23:10:45.673772 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.673341 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9wcc\" (UniqueName: \"kubernetes.io/projected/122b24bb-be61-4aa7-8faf-38f4c1f395f3-kube-api-access-x9wcc\") pod \"service-ca-operator-859f94c4b-8jk5k\" (UID: \"122b24bb-be61-4aa7-8faf-38f4c1f395f3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jk5k" Sep 09 23:10:45.673772 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.673390 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02b665a3-2525-4102-87eb-419606cea310-image-registry-private-configuration\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:45.673772 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.673433 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02b665a3-2525-4102-87eb-419606cea310-installation-pull-secrets\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:45.673772 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.673464 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-bound-sa-token\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:45.673772 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.673491 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/122b24bb-be61-4aa7-8faf-38f4c1f395f3-serving-cert\") pod \"service-ca-operator-859f94c4b-8jk5k\" (UID: \"122b24bb-be61-4aa7-8faf-38f4c1f395f3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jk5k" Sep 09 23:10:45.673772 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.673547 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7bxk\" (UniqueName: \"kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-kube-api-access-n7bxk\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:45.673772 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.673585 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a127828-4a97-40cf-b8ae-f5ecd20f97c6-config\") pod \"console-operator-85b454874b-2fs7f\" (UID: \"0a127828-4a97-40cf-b8ae-f5ecd20f97c6\") " pod="openshift-console-operator/console-operator-85b454874b-2fs7f" Sep 09 23:10:45.675605 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.675579 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6fa8702e-a63e-43aa-907c-e411e4af1b11-stats-auth\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:10:45.675605 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.675602 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6fa8702e-a63e-43aa-907c-e411e4af1b11-default-certificate\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:10:45.683051 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.683023 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq2v9\" (UniqueName: \"kubernetes.io/projected/6fa8702e-a63e-43aa-907c-e411e4af1b11-kube-api-access-jq2v9\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:10:45.774837 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.774782 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7bxk\" (UniqueName: \"kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-kube-api-access-n7bxk\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:45.774837 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.774838 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a127828-4a97-40cf-b8ae-f5ecd20f97c6-config\") pod \"console-operator-85b454874b-2fs7f\" (UID: \"0a127828-4a97-40cf-b8ae-f5ecd20f97c6\") " pod="openshift-console-operator/console-operator-85b454874b-2fs7f" Sep 09 23:10:45.775130 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.774984 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/122b24bb-be61-4aa7-8faf-38f4c1f395f3-config\") pod \"service-ca-operator-859f94c4b-8jk5k\" (UID: \"122b24bb-be61-4aa7-8faf-38f4c1f395f3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jk5k" Sep 09 23:10:45.775130 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.775013 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02b665a3-2525-4102-87eb-419606cea310-trusted-ca\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:45.775130 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.775033 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02b665a3-2525-4102-87eb-419606cea310-ca-trust-extracted\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:45.775130 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.775071 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a127828-4a97-40cf-b8ae-f5ecd20f97c6-serving-cert\") pod \"console-operator-85b454874b-2fs7f\" (UID: \"0a127828-4a97-40cf-b8ae-f5ecd20f97c6\") " pod="openshift-console-operator/console-operator-85b454874b-2fs7f" Sep 09 23:10:45.775130 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.775088 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjbl2\" (UniqueName: \"kubernetes.io/projected/0a127828-4a97-40cf-b8ae-f5ecd20f97c6-kube-api-access-sjbl2\") pod \"console-operator-85b454874b-2fs7f\" (UID: \"0a127828-4a97-40cf-b8ae-f5ecd20f97c6\") " pod="openshift-console-operator/console-operator-85b454874b-2fs7f" Sep 09 23:10:45.775130 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.775118 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02b665a3-2525-4102-87eb-419606cea310-registry-certificates\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:45.775460 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.775152 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkgn9\" (UniqueName: \"kubernetes.io/projected/72f66107-ac74-49f5-9782-583434006631-kube-api-access-qkgn9\") pod \"volume-data-source-validator-695c895bd6-rzk8q\" (UID: \"72f66107-ac74-49f5-9782-583434006631\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rzk8q" Sep 09 23:10:45.775460 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.775177 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-registry-tls\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:45.775460 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.775213 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a127828-4a97-40cf-b8ae-f5ecd20f97c6-trusted-ca\") pod \"console-operator-85b454874b-2fs7f\" (UID: \"0a127828-4a97-40cf-b8ae-f5ecd20f97c6\") " pod="openshift-console-operator/console-operator-85b454874b-2fs7f" Sep 09 23:10:45.775460 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.775241 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9wcc\" (UniqueName: \"kubernetes.io/projected/122b24bb-be61-4aa7-8faf-38f4c1f395f3-kube-api-access-x9wcc\") pod \"service-ca-operator-859f94c4b-8jk5k\" (UID: \"122b24bb-be61-4aa7-8faf-38f4c1f395f3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jk5k" Sep 09 23:10:45.775460 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.775284 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02b665a3-2525-4102-87eb-419606cea310-image-registry-private-configuration\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:45.775460 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.775315 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02b665a3-2525-4102-87eb-419606cea310-installation-pull-secrets\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:45.775460 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.775344 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-bound-sa-token\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:45.775460 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.775371 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/122b24bb-be61-4aa7-8faf-38f4c1f395f3-serving-cert\") pod \"service-ca-operator-859f94c4b-8jk5k\" (UID: \"122b24bb-be61-4aa7-8faf-38f4c1f395f3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jk5k" Sep 09 23:10:45.775460 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.775454 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02b665a3-2525-4102-87eb-419606cea310-ca-trust-extracted\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:45.776003 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.775628 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a127828-4a97-40cf-b8ae-f5ecd20f97c6-config\") pod \"console-operator-85b454874b-2fs7f\" (UID: \"0a127828-4a97-40cf-b8ae-f5ecd20f97c6\") " pod="openshift-console-operator/console-operator-85b454874b-2fs7f" Sep 09 23:10:45.776003 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.775677 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/122b24bb-be61-4aa7-8faf-38f4c1f395f3-config\") pod \"service-ca-operator-859f94c4b-8jk5k\" (UID: \"122b24bb-be61-4aa7-8faf-38f4c1f395f3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jk5k" Sep 09 23:10:45.776003 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:45.775767 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 23:10:45.776003 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:45.775782 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56cd69fd8c-bj9b8: secret "image-registry-tls" not found Sep 09 23:10:45.776003 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:45.775833 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-registry-tls podName:02b665a3-2525-4102-87eb-419606cea310 nodeName:}" failed. No retries permitted until 2026-09-09 23:10:46.275815843 +0000 UTC m=+234.961599242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-registry-tls") pod "image-registry-56cd69fd8c-bj9b8" (UID: "02b665a3-2525-4102-87eb-419606cea310") : secret "image-registry-tls" not found Sep 09 23:10:45.776438 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.776162 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02b665a3-2525-4102-87eb-419606cea310-registry-certificates\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:45.776554 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.776530 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a127828-4a97-40cf-b8ae-f5ecd20f97c6-trusted-ca\") pod \"console-operator-85b454874b-2fs7f\" (UID: \"0a127828-4a97-40cf-b8ae-f5ecd20f97c6\") " pod="openshift-console-operator/console-operator-85b454874b-2fs7f" Sep 09 23:10:45.776911 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.776576 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02b665a3-2525-4102-87eb-419606cea310-trusted-ca\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:45.777900 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.777877 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/122b24bb-be61-4aa7-8faf-38f4c1f395f3-serving-cert\") pod \"service-ca-operator-859f94c4b-8jk5k\" (UID: \"122b24bb-be61-4aa7-8faf-38f4c1f395f3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jk5k" Sep 09 23:10:45.777985 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.777969 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a127828-4a97-40cf-b8ae-f5ecd20f97c6-serving-cert\") pod \"console-operator-85b454874b-2fs7f\" (UID: \"0a127828-4a97-40cf-b8ae-f5ecd20f97c6\") " pod="openshift-console-operator/console-operator-85b454874b-2fs7f" Sep 09 23:10:45.778283 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.778266 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02b665a3-2525-4102-87eb-419606cea310-image-registry-private-configuration\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:45.778637 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.778621 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02b665a3-2525-4102-87eb-419606cea310-installation-pull-secrets\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:45.782490 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.782462 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjbl2\" (UniqueName: \"kubernetes.io/projected/0a127828-4a97-40cf-b8ae-f5ecd20f97c6-kube-api-access-sjbl2\") pod \"console-operator-85b454874b-2fs7f\" (UID: \"0a127828-4a97-40cf-b8ae-f5ecd20f97c6\") " pod="openshift-console-operator/console-operator-85b454874b-2fs7f" Sep 09 23:10:45.782568 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.782521 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7bxk\" (UniqueName: \"kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-kube-api-access-n7bxk\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:45.785560 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.785533 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-bound-sa-token\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:45.785807 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.785785 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9wcc\" (UniqueName: \"kubernetes.io/projected/122b24bb-be61-4aa7-8faf-38f4c1f395f3-kube-api-access-x9wcc\") pod \"service-ca-operator-859f94c4b-8jk5k\" (UID: \"122b24bb-be61-4aa7-8faf-38f4c1f395f3\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jk5k" Sep 09 23:10:45.785877 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.785822 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkgn9\" (UniqueName: \"kubernetes.io/projected/72f66107-ac74-49f5-9782-583434006631-kube-api-access-qkgn9\") pod \"volume-data-source-validator-695c895bd6-rzk8q\" (UID: \"72f66107-ac74-49f5-9782-583434006631\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rzk8q" Sep 09 23:10:45.819583 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.819551 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jk5k" Sep 09 23:10:45.829757 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.829739 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rzk8q" Sep 09 23:10:45.838895 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:45.838846 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-2fs7f" Sep 09 23:10:46.094911 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:46.094877 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-8jk5k"] Sep 09 23:10:46.097906 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:10:46.097879 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122b24bb_be61_4aa7_8faf_38f4c1f395f3.slice/crio-d348e5eb45ec8e4cea21bb4367cc60778e32936c4ee4ebe5fafd4386b2cc1923 WatchSource:0}: Error finding container d348e5eb45ec8e4cea21bb4367cc60778e32936c4ee4ebe5fafd4386b2cc1923: Status 404 returned error can't find the container with id d348e5eb45ec8e4cea21bb4367cc60778e32936c4ee4ebe5fafd4386b2cc1923 Sep 09 23:10:46.120922 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:46.120879 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rzk8q"] Sep 09 23:10:46.123893 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:46.123864 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-2fs7f"] Sep 09 23:10:46.124415 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:10:46.124385 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f66107_ac74_49f5_9782_583434006631.slice/crio-fc19f0be6c3a7afb19ddc6a99e2cbcc32ca5c90f09d766680c575d7cfd5940ac WatchSource:0}: Error finding container fc19f0be6c3a7afb19ddc6a99e2cbcc32ca5c90f09d766680c575d7cfd5940ac: Status 404 returned error can't find the container with id fc19f0be6c3a7afb19ddc6a99e2cbcc32ca5c90f09d766680c575d7cfd5940ac Sep 09 23:10:46.126518 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:10:46.126471 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a127828_4a97_40cf_b8ae_f5ecd20f97c6.slice/crio-a4a94dd7013948dc697347db0ecc4f25225991a3fb8a622af064bab1915ad9cf WatchSource:0}: Error finding container a4a94dd7013948dc697347db0ecc4f25225991a3fb8a622af064bab1915ad9cf: Status 404 returned error can't find the container with id a4a94dd7013948dc697347db0ecc4f25225991a3fb8a622af064bab1915ad9cf Sep 09 23:10:46.179812 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:46.179771 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa8702e-a63e-43aa-907c-e411e4af1b11-metrics-certs\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:10:46.179812 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:46.179832 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fa8702e-a63e-43aa-907c-e411e4af1b11-service-ca-bundle\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:10:46.180113 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:46.179976 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6fa8702e-a63e-43aa-907c-e411e4af1b11-service-ca-bundle podName:6fa8702e-a63e-43aa-907c-e411e4af1b11 nodeName:}" failed. No retries permitted until 2026-09-09 23:10:47.179957707 +0000 UTC m=+235.865741088 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6fa8702e-a63e-43aa-907c-e411e4af1b11-service-ca-bundle") pod "router-default-b46465886-mcs6c" (UID: "6fa8702e-a63e-43aa-907c-e411e4af1b11") : configmap references non-existent config key: service-ca.crt Sep 09 23:10:46.180113 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:46.179976 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 09 23:10:46.180113 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:46.180045 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fa8702e-a63e-43aa-907c-e411e4af1b11-metrics-certs podName:6fa8702e-a63e-43aa-907c-e411e4af1b11 nodeName:}" failed. No retries permitted until 2026-09-09 23:10:47.180026145 +0000 UTC m=+235.865809529 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fa8702e-a63e-43aa-907c-e411e4af1b11-metrics-certs") pod "router-default-b46465886-mcs6c" (UID: "6fa8702e-a63e-43aa-907c-e411e4af1b11") : secret "router-metrics-certs-default" not found Sep 09 23:10:46.280340 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:46.280296 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-registry-tls\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:46.280520 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:46.280441 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 23:10:46.280520 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:46.280459 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56cd69fd8c-bj9b8: secret "image-registry-tls" not found Sep 09 23:10:46.280520 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:46.280517 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-registry-tls podName:02b665a3-2525-4102-87eb-419606cea310 nodeName:}" failed. No retries permitted until 2026-09-09 23:10:47.280500782 +0000 UTC m=+235.966284162 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-registry-tls") pod "image-registry-56cd69fd8c-bj9b8" (UID: "02b665a3-2525-4102-87eb-419606cea310") : secret "image-registry-tls" not found Sep 09 23:10:46.597298 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:46.597207 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-2fs7f" event={"ID":"0a127828-4a97-40cf-b8ae-f5ecd20f97c6","Type":"ContainerStarted","Data":"a4a94dd7013948dc697347db0ecc4f25225991a3fb8a622af064bab1915ad9cf"} Sep 09 23:10:46.599191 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:46.599118 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rzk8q" event={"ID":"72f66107-ac74-49f5-9782-583434006631","Type":"ContainerStarted","Data":"fc19f0be6c3a7afb19ddc6a99e2cbcc32ca5c90f09d766680c575d7cfd5940ac"} Sep 09 23:10:46.601739 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:46.601673 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jk5k" event={"ID":"122b24bb-be61-4aa7-8faf-38f4c1f395f3","Type":"ContainerStarted","Data":"d348e5eb45ec8e4cea21bb4367cc60778e32936c4ee4ebe5fafd4386b2cc1923"} Sep 09 23:10:47.188205 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:47.188165 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa8702e-a63e-43aa-907c-e411e4af1b11-metrics-certs\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:10:47.188391 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:47.188234 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fa8702e-a63e-43aa-907c-e411e4af1b11-service-ca-bundle\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:10:47.188391 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:47.188378 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 09 23:10:47.188503 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:47.188426 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6fa8702e-a63e-43aa-907c-e411e4af1b11-service-ca-bundle podName:6fa8702e-a63e-43aa-907c-e411e4af1b11 nodeName:}" failed. No retries permitted until 2026-09-09 23:10:49.188403869 +0000 UTC m=+237.874187253 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6fa8702e-a63e-43aa-907c-e411e4af1b11-service-ca-bundle") pod "router-default-b46465886-mcs6c" (UID: "6fa8702e-a63e-43aa-907c-e411e4af1b11") : configmap references non-existent config key: service-ca.crt Sep 09 23:10:47.188503 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:47.188460 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fa8702e-a63e-43aa-907c-e411e4af1b11-metrics-certs podName:6fa8702e-a63e-43aa-907c-e411e4af1b11 nodeName:}" failed. No retries permitted until 2026-09-09 23:10:49.188449054 +0000 UTC m=+237.874232443 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fa8702e-a63e-43aa-907c-e411e4af1b11-metrics-certs") pod "router-default-b46465886-mcs6c" (UID: "6fa8702e-a63e-43aa-907c-e411e4af1b11") : secret "router-metrics-certs-default" not found Sep 09 23:10:47.289359 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:47.289324 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-registry-tls\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:47.289538 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:47.289519 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 23:10:47.289577 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:47.289544 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56cd69fd8c-bj9b8: secret "image-registry-tls" not found Sep 09 23:10:47.289625 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:47.289614 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-registry-tls podName:02b665a3-2525-4102-87eb-419606cea310 nodeName:}" failed. No retries permitted until 2026-09-09 23:10:49.289590637 +0000 UTC m=+237.975374019 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-registry-tls") pod "image-registry-56cd69fd8c-bj9b8" (UID: "02b665a3-2525-4102-87eb-419606cea310") : secret "image-registry-tls" not found Sep 09 23:10:47.606827 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:47.606738 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rzk8q" event={"ID":"72f66107-ac74-49f5-9782-583434006631","Type":"ContainerStarted","Data":"f634a757ebc2301c3d177c4be01afa9f2a38c4d4d1f49bc4d34b1e9be65e7903"} Sep 09 23:10:47.622699 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:47.622600 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-rzk8q" podStartSLOduration=1.518843653 podStartE2EDuration="2.622579613s" totalImagesPullingTime="1.10373596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:10:47.621192277 +0000 UTC m=+236.306975697" watchObservedRunningTime="2026-09-09 23:10:47.622579613 +0000 UTC m=+236.308363013" Sep 09 23:10:49.208217 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.208162 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fa8702e-a63e-43aa-907c-e411e4af1b11-service-ca-bundle\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:10:49.208612 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.208251 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa8702e-a63e-43aa-907c-e411e4af1b11-metrics-certs\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:10:49.208612 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:49.208344 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6fa8702e-a63e-43aa-907c-e411e4af1b11-service-ca-bundle podName:6fa8702e-a63e-43aa-907c-e411e4af1b11 nodeName:}" failed. No retries permitted until 2026-09-09 23:10:53.208325538 +0000 UTC m=+241.894108922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6fa8702e-a63e-43aa-907c-e411e4af1b11-service-ca-bundle") pod "router-default-b46465886-mcs6c" (UID: "6fa8702e-a63e-43aa-907c-e411e4af1b11") : configmap references non-existent config key: service-ca.crt Sep 09 23:10:49.208612 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:49.208354 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 09 23:10:49.208612 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:49.208400 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fa8702e-a63e-43aa-907c-e411e4af1b11-metrics-certs podName:6fa8702e-a63e-43aa-907c-e411e4af1b11 nodeName:}" failed. No retries permitted until 2026-09-09 23:10:53.208383535 +0000 UTC m=+241.894166917 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fa8702e-a63e-43aa-907c-e411e4af1b11-metrics-certs") pod "router-default-b46465886-mcs6c" (UID: "6fa8702e-a63e-43aa-907c-e411e4af1b11") : secret "router-metrics-certs-default" not found Sep 09 23:10:49.309641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.309608 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-registry-tls\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:49.309811 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:49.309730 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 23:10:49.309811 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:49.309742 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56cd69fd8c-bj9b8: secret "image-registry-tls" not found Sep 09 23:10:49.309811 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:49.309801 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-registry-tls podName:02b665a3-2525-4102-87eb-419606cea310 nodeName:}" failed. No retries permitted until 2026-09-09 23:10:53.309786132 +0000 UTC m=+241.995569514 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-registry-tls") pod "image-registry-56cd69fd8c-bj9b8" (UID: "02b665a3-2525-4102-87eb-419606cea310") : secret "image-registry-tls" not found Sep 09 23:10:49.614408 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.614332 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2fs7f_0a127828-4a97-40cf-b8ae-f5ecd20f97c6/console-operator/0.log" Sep 09 23:10:49.614408 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.614381 2561 generic.go:356] "Generic (PLEG): container finished" podID="0a127828-4a97-40cf-b8ae-f5ecd20f97c6" containerID="ee40ea4b9f6c33ac7a678e879c25131ea5f223c99ea60020d470a5d396da3c73" exitCode=255 Sep 09 23:10:49.614638 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.614468 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-2fs7f" event={"ID":"0a127828-4a97-40cf-b8ae-f5ecd20f97c6","Type":"ContainerDied","Data":"ee40ea4b9f6c33ac7a678e879c25131ea5f223c99ea60020d470a5d396da3c73"} Sep 09 23:10:49.614699 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.614673 2561 scope.go:117] "RemoveContainer" containerID="ee40ea4b9f6c33ac7a678e879c25131ea5f223c99ea60020d470a5d396da3c73" Sep 09 23:10:49.616168 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.616140 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jk5k" event={"ID":"122b24bb-be61-4aa7-8faf-38f4c1f395f3","Type":"ContainerStarted","Data":"150b6456faad98ff01dc7736c8f8ffd3f2c7b5741a86f27cf7311bf6965e41c6"} Sep 09 23:10:49.637641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.634979 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p2rqm"] Sep 09 23:10:49.641217 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.641193 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:10:49.643152 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.643129 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 09 23:10:49.643267 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.643229 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 09 23:10:49.643331 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.643315 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 09 23:10:49.643386 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.643339 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cdjwn\"" Sep 09 23:10:49.643444 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.643427 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 09 23:10:49.647000 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.646982 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p2rqm"] Sep 09 23:10:49.658979 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.658926 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jk5k" podStartSLOduration=2.185462902 podStartE2EDuration="4.658903578s" totalImagesPullingTime="2.473440676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:10:49.657799584 +0000 UTC m=+238.343582996" watchObservedRunningTime="2026-09-09 23:10:49.658903578 +0000 UTC m=+238.344686978" Sep 09 23:10:49.815037 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.814995 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b88ce14-8ff7-4f71-a165-62ccb17063ea-data-volume\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:10:49.815247 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.815048 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jch9b\" (UniqueName: \"kubernetes.io/projected/8b88ce14-8ff7-4f71-a165-62ccb17063ea-kube-api-access-jch9b\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:10:49.815247 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.815158 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b88ce14-8ff7-4f71-a165-62ccb17063ea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:10:49.815247 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.815196 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8b88ce14-8ff7-4f71-a165-62ccb17063ea-crio-socket\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:10:49.815426 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.815246 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b88ce14-8ff7-4f71-a165-62ccb17063ea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:10:49.915751 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.915656 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b88ce14-8ff7-4f71-a165-62ccb17063ea-data-volume\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:10:49.915751 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.915698 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jch9b\" (UniqueName: \"kubernetes.io/projected/8b88ce14-8ff7-4f71-a165-62ccb17063ea-kube-api-access-jch9b\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:10:49.915751 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.915740 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b88ce14-8ff7-4f71-a165-62ccb17063ea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:10:49.916011 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.915767 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8b88ce14-8ff7-4f71-a165-62ccb17063ea-crio-socket\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:10:49.916011 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.915813 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b88ce14-8ff7-4f71-a165-62ccb17063ea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:10:49.916011 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:49.915903 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 23:10:49.916011 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:49.915929 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:10:49.916011 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:49.915967 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8b88ce14-8ff7-4f71-a165-62ccb17063ea-kube-rbac-proxy-cm podName:8b88ce14-8ff7-4f71-a165-62ccb17063ea nodeName:}" failed. No retries permitted until 2026-09-09 23:10:50.415950385 +0000 UTC m=+239.101733786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8b88ce14-8ff7-4f71-a165-62ccb17063ea-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2rqm" (UID: "8b88ce14-8ff7-4f71-a165-62ccb17063ea") : configmap "kube-rbac-proxy" not found Sep 09 23:10:49.916011 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:49.915983 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b88ce14-8ff7-4f71-a165-62ccb17063ea-insights-runtime-extractor-tls podName:8b88ce14-8ff7-4f71-a165-62ccb17063ea nodeName:}" failed. No retries permitted until 2026-09-09 23:10:50.415977079 +0000 UTC m=+239.101760459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8b88ce14-8ff7-4f71-a165-62ccb17063ea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2rqm" (UID: "8b88ce14-8ff7-4f71-a165-62ccb17063ea") : secret "insights-runtime-extractor-tls" not found Sep 09 23:10:49.916011 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.915926 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8b88ce14-8ff7-4f71-a165-62ccb17063ea-crio-socket\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:10:49.916217 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.916084 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b88ce14-8ff7-4f71-a165-62ccb17063ea-data-volume\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:10:49.926130 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:49.926097 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jch9b\" (UniqueName: \"kubernetes.io/projected/8b88ce14-8ff7-4f71-a165-62ccb17063ea-kube-api-access-jch9b\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:10:50.418642 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:50.418599 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b88ce14-8ff7-4f71-a165-62ccb17063ea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:10:50.419125 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:50.418715 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b88ce14-8ff7-4f71-a165-62ccb17063ea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:10:50.419125 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:50.418790 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:10:50.419125 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:50.418801 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 23:10:50.419125 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:50.418895 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8b88ce14-8ff7-4f71-a165-62ccb17063ea-kube-rbac-proxy-cm podName:8b88ce14-8ff7-4f71-a165-62ccb17063ea nodeName:}" failed. No retries permitted until 2026-09-09 23:10:51.418874201 +0000 UTC m=+240.104657596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8b88ce14-8ff7-4f71-a165-62ccb17063ea-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2rqm" (UID: "8b88ce14-8ff7-4f71-a165-62ccb17063ea") : configmap "kube-rbac-proxy" not found Sep 09 23:10:50.419125 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:50.418926 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b88ce14-8ff7-4f71-a165-62ccb17063ea-insights-runtime-extractor-tls podName:8b88ce14-8ff7-4f71-a165-62ccb17063ea nodeName:}" failed. No retries permitted until 2026-09-09 23:10:51.418906059 +0000 UTC m=+240.104689460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8b88ce14-8ff7-4f71-a165-62ccb17063ea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2rqm" (UID: "8b88ce14-8ff7-4f71-a165-62ccb17063ea") : secret "insights-runtime-extractor-tls" not found Sep 09 23:10:50.621322 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:50.621294 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2fs7f_0a127828-4a97-40cf-b8ae-f5ecd20f97c6/console-operator/1.log" Sep 09 23:10:50.621729 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:50.621709 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2fs7f_0a127828-4a97-40cf-b8ae-f5ecd20f97c6/console-operator/0.log" Sep 09 23:10:50.621840 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:50.621755 2561 generic.go:356] "Generic (PLEG): container finished" podID="0a127828-4a97-40cf-b8ae-f5ecd20f97c6" containerID="fad7bbe9bc6b046308accd95648d25a507c9c233b1fff5da651bc28e82ca4588" exitCode=255 Sep 09 23:10:50.621928 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:50.621836 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-2fs7f" event={"ID":"0a127828-4a97-40cf-b8ae-f5ecd20f97c6","Type":"ContainerDied","Data":"fad7bbe9bc6b046308accd95648d25a507c9c233b1fff5da651bc28e82ca4588"} Sep 09 23:10:50.621928 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:50.621915 2561 scope.go:117] "RemoveContainer" containerID="ee40ea4b9f6c33ac7a678e879c25131ea5f223c99ea60020d470a5d396da3c73" Sep 09 23:10:50.622117 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:50.622095 2561 scope.go:117] "RemoveContainer" containerID="fad7bbe9bc6b046308accd95648d25a507c9c233b1fff5da651bc28e82ca4588" Sep 09 23:10:50.622289 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:50.622273 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-2fs7f_openshift-console-operator(0a127828-4a97-40cf-b8ae-f5ecd20f97c6)\"" pod="openshift-console-operator/console-operator-85b454874b-2fs7f" podUID="0a127828-4a97-40cf-b8ae-f5ecd20f97c6" Sep 09 23:10:51.427563 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:51.427523 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b88ce14-8ff7-4f71-a165-62ccb17063ea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:10:51.428162 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:51.427591 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b88ce14-8ff7-4f71-a165-62ccb17063ea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:10:51.428162 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:51.427652 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 23:10:51.428162 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:51.427728 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8b88ce14-8ff7-4f71-a165-62ccb17063ea-kube-rbac-proxy-cm podName:8b88ce14-8ff7-4f71-a165-62ccb17063ea nodeName:}" failed. No retries permitted until 2026-09-09 23:10:53.427714279 +0000 UTC m=+242.113497660 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8b88ce14-8ff7-4f71-a165-62ccb17063ea-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2rqm" (UID: "8b88ce14-8ff7-4f71-a165-62ccb17063ea") : configmap "kube-rbac-proxy" not found Sep 09 23:10:51.428162 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:51.427806 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:10:51.428162 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:51.427906 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b88ce14-8ff7-4f71-a165-62ccb17063ea-insights-runtime-extractor-tls podName:8b88ce14-8ff7-4f71-a165-62ccb17063ea nodeName:}" failed. No retries permitted until 2026-09-09 23:10:53.42788585 +0000 UTC m=+242.113669245 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8b88ce14-8ff7-4f71-a165-62ccb17063ea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2rqm" (UID: "8b88ce14-8ff7-4f71-a165-62ccb17063ea") : secret "insights-runtime-extractor-tls" not found Sep 09 23:10:51.590415 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:51.590386 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jqxbd_a843f1d8-0cd1-4396-8fc1-5a22cd20f6e8/dns-node-resolver/0.log" Sep 09 23:10:51.626507 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:51.626474 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2fs7f_0a127828-4a97-40cf-b8ae-f5ecd20f97c6/console-operator/1.log" Sep 09 23:10:51.626901 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:51.626881 2561 scope.go:117] "RemoveContainer" containerID="fad7bbe9bc6b046308accd95648d25a507c9c233b1fff5da651bc28e82ca4588" Sep 09 23:10:51.627069 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:51.627049 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-2fs7f_openshift-console-operator(0a127828-4a97-40cf-b8ae-f5ecd20f97c6)\"" pod="openshift-console-operator/console-operator-85b454874b-2fs7f" podUID="0a127828-4a97-40cf-b8ae-f5ecd20f97c6" Sep 09 23:10:52.257402 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:52.257357 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-68mgg"] Sep 09 23:10:52.261349 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:52.261324 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-68mgg" Sep 09 23:10:52.263707 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:52.263687 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 09 23:10:52.263707 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:52.263697 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 09 23:10:52.263928 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:52.263799 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 09 23:10:52.264017 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:52.264005 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 09 23:10:52.264735 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:52.264722 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zxkcb\"" Sep 09 23:10:52.266923 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:52.266903 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-68mgg"] Sep 09 23:10:52.436880 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:52.436822 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4c9e8314-cc76-41b7-8b58-a71315bd8f7a-signing-key\") pod \"service-ca-75d99c484-68mgg\" (UID: \"4c9e8314-cc76-41b7-8b58-a71315bd8f7a\") " pod="openshift-service-ca/service-ca-75d99c484-68mgg" Sep 09 23:10:52.437326 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:52.436898 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4c9e8314-cc76-41b7-8b58-a71315bd8f7a-signing-cabundle\") pod \"service-ca-75d99c484-68mgg\" (UID: \"4c9e8314-cc76-41b7-8b58-a71315bd8f7a\") " pod="openshift-service-ca/service-ca-75d99c484-68mgg" Sep 09 23:10:52.437326 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:52.436939 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rqmj\" (UniqueName: \"kubernetes.io/projected/4c9e8314-cc76-41b7-8b58-a71315bd8f7a-kube-api-access-2rqmj\") pod \"service-ca-75d99c484-68mgg\" (UID: \"4c9e8314-cc76-41b7-8b58-a71315bd8f7a\") " pod="openshift-service-ca/service-ca-75d99c484-68mgg" Sep 09 23:10:52.538060 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:52.537945 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4c9e8314-cc76-41b7-8b58-a71315bd8f7a-signing-key\") pod \"service-ca-75d99c484-68mgg\" (UID: \"4c9e8314-cc76-41b7-8b58-a71315bd8f7a\") " pod="openshift-service-ca/service-ca-75d99c484-68mgg" Sep 09 23:10:52.538060 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:52.538011 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4c9e8314-cc76-41b7-8b58-a71315bd8f7a-signing-cabundle\") pod \"service-ca-75d99c484-68mgg\" (UID: \"4c9e8314-cc76-41b7-8b58-a71315bd8f7a\") " pod="openshift-service-ca/service-ca-75d99c484-68mgg" Sep 09 23:10:52.538060 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:52.538050 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rqmj\" (UniqueName: \"kubernetes.io/projected/4c9e8314-cc76-41b7-8b58-a71315bd8f7a-kube-api-access-2rqmj\") pod \"service-ca-75d99c484-68mgg\" (UID: \"4c9e8314-cc76-41b7-8b58-a71315bd8f7a\") " pod="openshift-service-ca/service-ca-75d99c484-68mgg" Sep 09 23:10:52.538575 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:52.538552 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4c9e8314-cc76-41b7-8b58-a71315bd8f7a-signing-cabundle\") pod \"service-ca-75d99c484-68mgg\" (UID: \"4c9e8314-cc76-41b7-8b58-a71315bd8f7a\") " pod="openshift-service-ca/service-ca-75d99c484-68mgg" Sep 09 23:10:52.540574 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:52.540558 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4c9e8314-cc76-41b7-8b58-a71315bd8f7a-signing-key\") pod \"service-ca-75d99c484-68mgg\" (UID: \"4c9e8314-cc76-41b7-8b58-a71315bd8f7a\") " pod="openshift-service-ca/service-ca-75d99c484-68mgg" Sep 09 23:10:52.545740 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:52.545713 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rqmj\" (UniqueName: \"kubernetes.io/projected/4c9e8314-cc76-41b7-8b58-a71315bd8f7a-kube-api-access-2rqmj\") pod \"service-ca-75d99c484-68mgg\" (UID: \"4c9e8314-cc76-41b7-8b58-a71315bd8f7a\") " pod="openshift-service-ca/service-ca-75d99c484-68mgg" Sep 09 23:10:52.570719 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:52.570682 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-68mgg" Sep 09 23:10:52.588451 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:52.588426 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d76sb_d702c7a3-03a5-4b8b-9a7f-df843575cc43/node-ca/0.log" Sep 09 23:10:52.842421 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:52.842350 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-68mgg"] Sep 09 23:10:52.845077 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:10:52.845050 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9e8314_cc76_41b7_8b58_a71315bd8f7a.slice/crio-b4358633996b09bdaf2c721d1b9ccc9c24614ad881477d869108f048e48c2852 WatchSource:0}: Error finding container b4358633996b09bdaf2c721d1b9ccc9c24614ad881477d869108f048e48c2852: Status 404 returned error can't find the container with id b4358633996b09bdaf2c721d1b9ccc9c24614ad881477d869108f048e48c2852 Sep 09 23:10:53.244813 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:53.244758 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa8702e-a63e-43aa-907c-e411e4af1b11-metrics-certs\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:10:53.245037 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:53.244845 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fa8702e-a63e-43aa-907c-e411e4af1b11-service-ca-bundle\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:10:53.245037 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:53.244945 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 09 23:10:53.245037 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:53.245019 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fa8702e-a63e-43aa-907c-e411e4af1b11-metrics-certs podName:6fa8702e-a63e-43aa-907c-e411e4af1b11 nodeName:}" failed. No retries permitted until 2026-09-09 23:11:01.244997919 +0000 UTC m=+249.930781301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6fa8702e-a63e-43aa-907c-e411e4af1b11-metrics-certs") pod "router-default-b46465886-mcs6c" (UID: "6fa8702e-a63e-43aa-907c-e411e4af1b11") : secret "router-metrics-certs-default" not found Sep 09 23:10:53.245037 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:53.245036 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6fa8702e-a63e-43aa-907c-e411e4af1b11-service-ca-bundle podName:6fa8702e-a63e-43aa-907c-e411e4af1b11 nodeName:}" failed. No retries permitted until 2026-09-09 23:11:01.245029954 +0000 UTC m=+249.930813335 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6fa8702e-a63e-43aa-907c-e411e4af1b11-service-ca-bundle") pod "router-default-b46465886-mcs6c" (UID: "6fa8702e-a63e-43aa-907c-e411e4af1b11") : configmap references non-existent config key: service-ca.crt Sep 09 23:10:53.345681 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:53.345639 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-registry-tls\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:10:53.345904 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:53.345796 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 23:10:53.345904 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:53.345819 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56cd69fd8c-bj9b8: secret "image-registry-tls" not found Sep 09 23:10:53.345904 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:53.345904 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-registry-tls podName:02b665a3-2525-4102-87eb-419606cea310 nodeName:}" failed. No retries permitted until 2026-09-09 23:11:01.345882244 +0000 UTC m=+250.031665642 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-registry-tls") pod "image-registry-56cd69fd8c-bj9b8" (UID: "02b665a3-2525-4102-87eb-419606cea310") : secret "image-registry-tls" not found Sep 09 23:10:53.446839 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:53.446794 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b88ce14-8ff7-4f71-a165-62ccb17063ea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:10:53.447279 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:53.446890 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b88ce14-8ff7-4f71-a165-62ccb17063ea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:10:53.447279 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:53.446948 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 23:10:53.447279 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:53.447026 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8b88ce14-8ff7-4f71-a165-62ccb17063ea-kube-rbac-proxy-cm podName:8b88ce14-8ff7-4f71-a165-62ccb17063ea nodeName:}" failed. No retries permitted until 2026-09-09 23:10:57.447005446 +0000 UTC m=+246.132788849 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8b88ce14-8ff7-4f71-a165-62ccb17063ea-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2rqm" (UID: "8b88ce14-8ff7-4f71-a165-62ccb17063ea") : configmap "kube-rbac-proxy" not found Sep 09 23:10:53.447279 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:53.447039 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:10:53.447279 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:53.447087 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b88ce14-8ff7-4f71-a165-62ccb17063ea-insights-runtime-extractor-tls podName:8b88ce14-8ff7-4f71-a165-62ccb17063ea nodeName:}" failed. No retries permitted until 2026-09-09 23:10:57.447074293 +0000 UTC m=+246.132857679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8b88ce14-8ff7-4f71-a165-62ccb17063ea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2rqm" (UID: "8b88ce14-8ff7-4f71-a165-62ccb17063ea") : secret "insights-runtime-extractor-tls" not found Sep 09 23:10:53.588367 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:53.588287 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xbplg_e060ec1e-b4e7-4369-9387-b6817a2bcd4a/serve-healthcheck-canary/0.log" Sep 09 23:10:53.635896 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:53.635833 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-68mgg" event={"ID":"4c9e8314-cc76-41b7-8b58-a71315bd8f7a","Type":"ContainerStarted","Data":"e7bf1f943e90d08cd8720e3176b5966ec2257721fa95816d57d5d6ddc857df08"} Sep 09 23:10:53.636060 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:53.635919 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-68mgg" event={"ID":"4c9e8314-cc76-41b7-8b58-a71315bd8f7a","Type":"ContainerStarted","Data":"b4358633996b09bdaf2c721d1b9ccc9c24614ad881477d869108f048e48c2852"} Sep 09 23:10:53.650804 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:53.650746 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-68mgg" podStartSLOduration=1.650729015 podStartE2EDuration="1.650729015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:10:53.650076217 +0000 UTC m=+242.335859642" watchObservedRunningTime="2026-09-09 23:10:53.650729015 +0000 UTC m=+242.336512417" Sep 09 23:10:55.839493 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:55.839455 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-2fs7f" Sep 09 23:10:55.840006 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:55.839523 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-2fs7f" Sep 09 23:10:55.840006 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:55.839973 2561 scope.go:117] "RemoveContainer" containerID="fad7bbe9bc6b046308accd95648d25a507c9c233b1fff5da651bc28e82ca4588" Sep 09 23:10:55.840182 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:55.840161 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-2fs7f_openshift-console-operator(0a127828-4a97-40cf-b8ae-f5ecd20f97c6)\"" pod="openshift-console-operator/console-operator-85b454874b-2fs7f" podUID="0a127828-4a97-40cf-b8ae-f5ecd20f97c6" Sep 09 23:10:57.480771 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:57.480725 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b88ce14-8ff7-4f71-a165-62ccb17063ea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:10:57.481269 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:57.480783 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b88ce14-8ff7-4f71-a165-62ccb17063ea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:10:57.481269 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:57.480876 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 23:10:57.481269 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:57.480912 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:10:57.481269 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:57.480942 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8b88ce14-8ff7-4f71-a165-62ccb17063ea-kube-rbac-proxy-cm podName:8b88ce14-8ff7-4f71-a165-62ccb17063ea nodeName:}" failed. No retries permitted until 2026-09-09 23:11:05.480925793 +0000 UTC m=+254.166709174 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8b88ce14-8ff7-4f71-a165-62ccb17063ea-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2rqm" (UID: "8b88ce14-8ff7-4f71-a165-62ccb17063ea") : configmap "kube-rbac-proxy" not found Sep 09 23:10:57.481269 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:10:57.480957 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b88ce14-8ff7-4f71-a165-62ccb17063ea-insights-runtime-extractor-tls podName:8b88ce14-8ff7-4f71-a165-62ccb17063ea nodeName:}" failed. No retries permitted until 2026-09-09 23:11:05.480950815 +0000 UTC m=+254.166734195 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8b88ce14-8ff7-4f71-a165-62ccb17063ea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2rqm" (UID: "8b88ce14-8ff7-4f71-a165-62ccb17063ea") : secret "insights-runtime-extractor-tls" not found Sep 09 23:10:59.659403 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:59.659312 2561 generic.go:356] "Generic (PLEG): container finished" podID="02ac530c-6761-4265-b864-6af6f318b272" containerID="a9ce785ed81808596ecad0e0006a31ddfcf43ab27820731c31acc43d37c30e86" exitCode=255 Sep 09 23:10:59.659403 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:59.659389 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579b867994-qz5gz" event={"ID":"02ac530c-6761-4265-b864-6af6f318b272","Type":"ContainerDied","Data":"a9ce785ed81808596ecad0e0006a31ddfcf43ab27820731c31acc43d37c30e86"} Sep 09 23:10:59.665342 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:10:59.665319 2561 scope.go:117] "RemoveContainer" containerID="a9ce785ed81808596ecad0e0006a31ddfcf43ab27820731c31acc43d37c30e86" Sep 09 23:11:00.664403 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:00.664368 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579b867994-qz5gz" event={"ID":"02ac530c-6761-4265-b864-6af6f318b272","Type":"ContainerStarted","Data":"f988b39b025a8f73efc214b126e3c1ec40ad6ca74740f035be8b4ab4459a3d8d"} Sep 09 23:11:01.316954 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:01.316916 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa8702e-a63e-43aa-907c-e411e4af1b11-metrics-certs\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:11:01.317166 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:01.316989 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fa8702e-a63e-43aa-907c-e411e4af1b11-service-ca-bundle\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:11:01.317224 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:11:01.317158 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6fa8702e-a63e-43aa-907c-e411e4af1b11-service-ca-bundle podName:6fa8702e-a63e-43aa-907c-e411e4af1b11 nodeName:}" failed. No retries permitted until 2026-09-09 23:11:17.317136 +0000 UTC m=+266.002919397 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6fa8702e-a63e-43aa-907c-e411e4af1b11-service-ca-bundle") pod "router-default-b46465886-mcs6c" (UID: "6fa8702e-a63e-43aa-907c-e411e4af1b11") : configmap references non-existent config key: service-ca.crt Sep 09 23:11:01.319731 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:01.319697 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa8702e-a63e-43aa-907c-e411e4af1b11-metrics-certs\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:11:01.417617 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:01.417565 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-registry-tls\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:11:01.420178 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:01.420152 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-registry-tls\") pod \"image-registry-56cd69fd8c-bj9b8\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:11:01.446038 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:01.446010 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p9f5w\"" Sep 09 23:11:01.454806 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:01.454783 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:11:01.725613 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:01.725578 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56cd69fd8c-bj9b8"] Sep 09 23:11:01.728830 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:11:01.728798 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b665a3_2525_4102_87eb_419606cea310.slice/crio-b3d1a68f691482aaebb07fd22c976519c1026afde885e8f7f683ffa7af33cba3 WatchSource:0}: Error finding container b3d1a68f691482aaebb07fd22c976519c1026afde885e8f7f683ffa7af33cba3: Status 404 returned error can't find the container with id b3d1a68f691482aaebb07fd22c976519c1026afde885e8f7f683ffa7af33cba3 Sep 09 23:11:02.672155 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:02.672119 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" event={"ID":"02b665a3-2525-4102-87eb-419606cea310","Type":"ContainerStarted","Data":"5ce237c86a10e70fc1a1b971094354695adc61710049540b69128827dd589498"} Sep 09 23:11:02.672358 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:02.672196 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" event={"ID":"02b665a3-2525-4102-87eb-419606cea310","Type":"ContainerStarted","Data":"b3d1a68f691482aaebb07fd22c976519c1026afde885e8f7f683ffa7af33cba3"} Sep 09 23:11:02.672358 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:02.672285 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:11:02.690415 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:02.690358 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" podStartSLOduration=17.690341298 podStartE2EDuration="17.690341298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:11:02.689279479 +0000 UTC m=+251.375062882" watchObservedRunningTime="2026-09-09 23:11:02.690341298 +0000 UTC m=+251.376124700" Sep 09 23:11:03.635869 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:03.635817 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs\") pod \"network-metrics-daemon-j89rl\" (UID: \"9e224f1b-9ce8-484e-8de5-f4f4f49a5eab\") " pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:11:03.638892 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:03.638830 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e224f1b-9ce8-484e-8de5-f4f4f49a5eab-metrics-certs\") pod \"network-metrics-daemon-j89rl\" (UID: \"9e224f1b-9ce8-484e-8de5-f4f4f49a5eab\") " pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:11:03.707779 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:03.707748 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h952m\"" Sep 09 23:11:03.716654 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:03.716631 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j89rl" Sep 09 23:11:03.986885 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:03.986836 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j89rl"] Sep 09 23:11:03.989919 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:11:03.989891 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e224f1b_9ce8_484e_8de5_f4f4f49a5eab.slice/crio-58b69fc33031412934c3516ee2ddd674db667e2ebe86a29fa19cc15804b8b2a3 WatchSource:0}: Error finding container 58b69fc33031412934c3516ee2ddd674db667e2ebe86a29fa19cc15804b8b2a3: Status 404 returned error can't find the container with id 58b69fc33031412934c3516ee2ddd674db667e2ebe86a29fa19cc15804b8b2a3 Sep 09 23:11:04.679935 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:04.679900 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j89rl" event={"ID":"9e224f1b-9ce8-484e-8de5-f4f4f49a5eab","Type":"ContainerStarted","Data":"58b69fc33031412934c3516ee2ddd674db667e2ebe86a29fa19cc15804b8b2a3"} Sep 09 23:11:05.550177 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:05.550065 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b88ce14-8ff7-4f71-a165-62ccb17063ea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:11:05.550177 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:05.550126 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b88ce14-8ff7-4f71-a165-62ccb17063ea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:11:05.550377 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:11:05.550231 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:11:05.550377 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:11:05.550310 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b88ce14-8ff7-4f71-a165-62ccb17063ea-insights-runtime-extractor-tls podName:8b88ce14-8ff7-4f71-a165-62ccb17063ea nodeName:}" failed. No retries permitted until 2026-09-09 23:11:21.550289565 +0000 UTC m=+270.236072959 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8b88ce14-8ff7-4f71-a165-62ccb17063ea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2rqm" (UID: "8b88ce14-8ff7-4f71-a165-62ccb17063ea") : secret "insights-runtime-extractor-tls" not found Sep 09 23:11:05.550765 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:05.550745 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b88ce14-8ff7-4f71-a165-62ccb17063ea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:11:05.687893 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:05.687843 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j89rl" event={"ID":"9e224f1b-9ce8-484e-8de5-f4f4f49a5eab","Type":"ContainerStarted","Data":"bcd4a86e874f45669fb55c28808ff8c0ee05faeaa23ace55518f64d02a6b73a6"} Sep 09 23:11:05.688290 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:05.687917 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j89rl" event={"ID":"9e224f1b-9ce8-484e-8de5-f4f4f49a5eab","Type":"ContainerStarted","Data":"4587171d6452f5a5f251a708f4fa629cc9db9b57aca15b43af1e1c539b3e2f8d"} Sep 09 23:11:05.701970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:05.701918 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-j89rl" podStartSLOduration=252.442192948 podStartE2EDuration="4m13.701900803s" totalImagesPullingTime="1.259707855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:11:05.701012239 +0000 UTC m=+254.386795657" watchObservedRunningTime="2026-09-09 23:11:05.701900803 +0000 UTC m=+254.387684218" Sep 09 23:11:07.808516 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:07.808479 2561 scope.go:117] "RemoveContainer" containerID="fad7bbe9bc6b046308accd95648d25a507c9c233b1fff5da651bc28e82ca4588" Sep 09 23:11:08.704275 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:08.704236 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2fs7f_0a127828-4a97-40cf-b8ae-f5ecd20f97c6/console-operator/1.log" Sep 09 23:11:08.704427 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:08.704337 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-2fs7f" event={"ID":"0a127828-4a97-40cf-b8ae-f5ecd20f97c6","Type":"ContainerStarted","Data":"a51e66fe98aae45585325dbe96e9771de6deddbe2526b0659eca4bf1506a226d"} Sep 09 23:11:08.704655 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:08.704629 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-2fs7f" Sep 09 23:11:08.709359 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:08.709335 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-2fs7f" Sep 09 23:11:08.720753 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:08.720697 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-2fs7f" podStartSLOduration=21.280736447 podStartE2EDuration="23.720682741s" totalImagesPullingTime="2.439946294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:11:08.719374305 +0000 UTC m=+257.405157724" watchObservedRunningTime="2026-09-09 23:11:08.720682741 +0000 UTC m=+257.406466183" Sep 09 23:11:12.746505 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:12.746422 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56cd69fd8c-bj9b8"] Sep 09 23:11:17.341003 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:17.340957 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fa8702e-a63e-43aa-907c-e411e4af1b11-service-ca-bundle\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:11:17.341645 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:17.341620 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fa8702e-a63e-43aa-907c-e411e4af1b11-service-ca-bundle\") pod \"router-default-b46465886-mcs6c\" (UID: \"6fa8702e-a63e-43aa-907c-e411e4af1b11\") " pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:11:17.535803 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:17.535772 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-n9ws5\"" Sep 09 23:11:17.544171 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:17.544148 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:11:17.846613 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:17.846574 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b46465886-mcs6c"] Sep 09 23:11:17.849288 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:11:17.849260 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa8702e_a63e_43aa_907c_e411e4af1b11.slice/crio-25d1e1f59cc85f6d1428393eb7e01d46acb8d2a0fd8161b069700782c28308ba WatchSource:0}: Error finding container 25d1e1f59cc85f6d1428393eb7e01d46acb8d2a0fd8161b069700782c28308ba: Status 404 returned error can't find the container with id 25d1e1f59cc85f6d1428393eb7e01d46acb8d2a0fd8161b069700782c28308ba Sep 09 23:11:18.744627 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:18.744594 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b46465886-mcs6c" event={"ID":"6fa8702e-a63e-43aa-907c-e411e4af1b11","Type":"ContainerStarted","Data":"a533ad7ae0bdf6ff159aca379b2eafd3c48ab7b059253113f3b087ec9a51d0a0"} Sep 09 23:11:18.745034 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:18.744648 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b46465886-mcs6c" event={"ID":"6fa8702e-a63e-43aa-907c-e411e4af1b11","Type":"ContainerStarted","Data":"25d1e1f59cc85f6d1428393eb7e01d46acb8d2a0fd8161b069700782c28308ba"} Sep 09 23:11:18.762079 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:18.762025 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-b46465886-mcs6c" podStartSLOduration=33.762007636 podStartE2EDuration="33.762007636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:11:18.760884302 +0000 UTC m=+267.446667718" watchObservedRunningTime="2026-09-09 23:11:18.762007636 +0000 UTC m=+267.447791038" Sep 09 23:11:19.544648 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:19.544600 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:11:19.547662 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:19.547635 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:11:19.748378 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:19.748340 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:11:19.749667 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:19.749643 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-b46465886-mcs6c" Sep 09 23:11:21.573232 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:21.573185 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b88ce14-8ff7-4f71-a165-62ccb17063ea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:11:21.575710 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:21.575688 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b88ce14-8ff7-4f71-a165-62ccb17063ea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2rqm\" (UID: \"8b88ce14-8ff7-4f71-a165-62ccb17063ea\") " pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:11:21.770543 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:21.770508 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cdjwn\"" Sep 09 23:11:21.778383 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:21.778345 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p2rqm" Sep 09 23:11:22.053824 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:22.053793 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p2rqm"] Sep 09 23:11:22.058090 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:11:22.058054 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b88ce14_8ff7_4f71_a165_62ccb17063ea.slice/crio-000f9a5adf2c736cc64f95f70edca7c4571885b8b06bb22ae158edcb58fbf062 WatchSource:0}: Error finding container 000f9a5adf2c736cc64f95f70edca7c4571885b8b06bb22ae158edcb58fbf062: Status 404 returned error can't find the container with id 000f9a5adf2c736cc64f95f70edca7c4571885b8b06bb22ae158edcb58fbf062 Sep 09 23:11:22.753922 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:22.753885 2561 patch_prober.go:28] interesting pod/image-registry-56cd69fd8c-bj9b8 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 09 23:11:22.754347 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:22.753984 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" podUID="02b665a3-2525-4102-87eb-419606cea310" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 09 23:11:22.760355 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:22.760324 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2rqm" event={"ID":"8b88ce14-8ff7-4f71-a165-62ccb17063ea","Type":"ContainerStarted","Data":"7a519f20070515625761474bedabe8233c9f4290a0e73be39fbbf7fd73a0f88b"} Sep 09 23:11:22.760486 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:22.760385 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2rqm" event={"ID":"8b88ce14-8ff7-4f71-a165-62ccb17063ea","Type":"ContainerStarted","Data":"000f9a5adf2c736cc64f95f70edca7c4571885b8b06bb22ae158edcb58fbf062"} Sep 09 23:11:23.268401 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.268367 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fd4b45f9b-lgxgd"] Sep 09 23:11:23.271623 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.271597 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:11:23.273904 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.273871 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 09 23:11:23.274049 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.274001 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 09 23:11:23.274778 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.274695 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 09 23:11:23.274778 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.274709 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4cpxx\"" Sep 09 23:11:23.274778 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.274774 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 09 23:11:23.275012 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.274842 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 09 23:11:23.275012 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.274935 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 09 23:11:23.275109 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.275028 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 09 23:11:23.286517 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.283127 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fd4b45f9b-lgxgd"] Sep 09 23:11:23.286517 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.284959 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 09 23:11:23.384916 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.384873 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/612bca39-7119-4574-ad6b-16ae0adc95e4-service-ca\") pod \"console-5fd4b45f9b-lgxgd\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:11:23.385108 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.384988 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/612bca39-7119-4574-ad6b-16ae0adc95e4-console-serving-cert\") pod \"console-5fd4b45f9b-lgxgd\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:11:23.385108 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.385041 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrfm4\" (UniqueName: \"kubernetes.io/projected/612bca39-7119-4574-ad6b-16ae0adc95e4-kube-api-access-qrfm4\") pod \"console-5fd4b45f9b-lgxgd\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:11:23.385108 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.385065 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/612bca39-7119-4574-ad6b-16ae0adc95e4-trusted-ca-bundle\") pod \"console-5fd4b45f9b-lgxgd\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:11:23.385108 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.385095 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/612bca39-7119-4574-ad6b-16ae0adc95e4-oauth-serving-cert\") pod \"console-5fd4b45f9b-lgxgd\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:11:23.385315 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.385141 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/612bca39-7119-4574-ad6b-16ae0adc95e4-console-config\") pod \"console-5fd4b45f9b-lgxgd\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:11:23.385315 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.385276 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/612bca39-7119-4574-ad6b-16ae0adc95e4-console-oauth-config\") pod \"console-5fd4b45f9b-lgxgd\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:11:23.486266 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.486224 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/612bca39-7119-4574-ad6b-16ae0adc95e4-console-oauth-config\") pod \"console-5fd4b45f9b-lgxgd\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:11:23.486473 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.486281 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/612bca39-7119-4574-ad6b-16ae0adc95e4-service-ca\") pod \"console-5fd4b45f9b-lgxgd\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:11:23.486473 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.486319 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/612bca39-7119-4574-ad6b-16ae0adc95e4-console-serving-cert\") pod \"console-5fd4b45f9b-lgxgd\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:11:23.486473 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.486350 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrfm4\" (UniqueName: \"kubernetes.io/projected/612bca39-7119-4574-ad6b-16ae0adc95e4-kube-api-access-qrfm4\") pod \"console-5fd4b45f9b-lgxgd\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:11:23.486473 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.486380 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/612bca39-7119-4574-ad6b-16ae0adc95e4-trusted-ca-bundle\") pod \"console-5fd4b45f9b-lgxgd\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:11:23.486473 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.486402 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/612bca39-7119-4574-ad6b-16ae0adc95e4-oauth-serving-cert\") pod \"console-5fd4b45f9b-lgxgd\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:11:23.486473 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.486427 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/612bca39-7119-4574-ad6b-16ae0adc95e4-console-config\") pod \"console-5fd4b45f9b-lgxgd\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:11:23.487272 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.487242 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/612bca39-7119-4574-ad6b-16ae0adc95e4-console-config\") pod \"console-5fd4b45f9b-lgxgd\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:11:23.487408 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.487338 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/612bca39-7119-4574-ad6b-16ae0adc95e4-oauth-serving-cert\") pod \"console-5fd4b45f9b-lgxgd\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:11:23.487488 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.487443 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/612bca39-7119-4574-ad6b-16ae0adc95e4-service-ca\") pod \"console-5fd4b45f9b-lgxgd\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:11:23.487573 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.487545 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/612bca39-7119-4574-ad6b-16ae0adc95e4-trusted-ca-bundle\") pod \"console-5fd4b45f9b-lgxgd\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:11:23.489192 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.489169 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/612bca39-7119-4574-ad6b-16ae0adc95e4-console-oauth-config\") pod \"console-5fd4b45f9b-lgxgd\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:11:23.489549 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.489513 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/612bca39-7119-4574-ad6b-16ae0adc95e4-console-serving-cert\") pod \"console-5fd4b45f9b-lgxgd\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:11:23.493930 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.493908 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrfm4\" (UniqueName: \"kubernetes.io/projected/612bca39-7119-4574-ad6b-16ae0adc95e4-kube-api-access-qrfm4\") pod \"console-5fd4b45f9b-lgxgd\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:11:23.583137 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.583043 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:11:23.766895 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.766841 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2rqm" event={"ID":"8b88ce14-8ff7-4f71-a165-62ccb17063ea","Type":"ContainerStarted","Data":"cbe681ed4c3198e6b65bbf961dab8277b183288a542de0fd6a56d560880a9996"} Sep 09 23:11:23.864090 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:23.864002 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fd4b45f9b-lgxgd"] Sep 09 23:11:24.170582 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:11:24.170507 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612bca39_7119_4574_ad6b_16ae0adc95e4.slice/crio-4c546fde3621b1858c168667eece3b8ac6f044b335726c1c1439ae2590b4d9d2 WatchSource:0}: Error finding container 4c546fde3621b1858c168667eece3b8ac6f044b335726c1c1439ae2590b4d9d2: Status 404 returned error can't find the container with id 4c546fde3621b1858c168667eece3b8ac6f044b335726c1c1439ae2590b4d9d2 Sep 09 23:11:24.773137 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:24.773083 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2rqm" event={"ID":"8b88ce14-8ff7-4f71-a165-62ccb17063ea","Type":"ContainerStarted","Data":"3d43e63dceae04151e9fc56dc7881f904d0c69b63704aa301f53b5dcef66d17a"} Sep 09 23:11:24.774810 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:24.774781 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fd4b45f9b-lgxgd" event={"ID":"612bca39-7119-4574-ad6b-16ae0adc95e4","Type":"ContainerStarted","Data":"4c546fde3621b1858c168667eece3b8ac6f044b335726c1c1439ae2590b4d9d2"} Sep 09 23:11:24.789679 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:24.789616 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-p2rqm" podStartSLOduration=33.743881644 podStartE2EDuration="35.789598812s" totalImagesPullingTime="2.045717168s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:10:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:11:24.787958786 +0000 UTC m=+273.473742217" watchObservedRunningTime="2026-09-09 23:11:24.789598812 +0000 UTC m=+273.475382224" Sep 09 23:11:27.786417 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:27.786378 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fd4b45f9b-lgxgd" event={"ID":"612bca39-7119-4574-ad6b-16ae0adc95e4","Type":"ContainerStarted","Data":"6c44fa49728029a607f4986c5832966b3b02c6b6f3b62c36c15dd6e75c1bf433"} Sep 09 23:11:27.803760 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:27.803710 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5fd4b45f9b-lgxgd" podStartSLOduration=2.155371317 podStartE2EDuration="4.803694253s" totalImagesPullingTime="2.648322936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:11:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:11:27.802263907 +0000 UTC m=+276.488047321" watchObservedRunningTime="2026-09-09 23:11:27.803694253 +0000 UTC m=+276.489477656" Sep 09 23:11:32.406728 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:11:32.406681 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-tdvkg" podUID="e9be36cc-dc4f-414f-98cb-ce27d9d70ff2" Sep 09 23:11:32.751486 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:32.751455 2561 patch_prober.go:28] interesting pod/image-registry-56cd69fd8c-bj9b8 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 09 23:11:32.751649 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:32.751504 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" podUID="02b665a3-2525-4102-87eb-419606cea310" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 09 23:11:32.803360 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:32.803326 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tdvkg" Sep 09 23:11:33.583325 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:33.583291 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:11:33.583768 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:33.583353 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:11:33.584742 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:33.584720 2561 patch_prober.go:28] interesting pod/console-5fd4b45f9b-lgxgd container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.16:8443/health\": dial tcp 10.134.0.16:8443: connect: connection refused" start-of-body= Sep 09 23:11:33.584817 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:33.584762 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5fd4b45f9b-lgxgd" podUID="612bca39-7119-4574-ad6b-16ae0adc95e4" containerName="console" probeResult="failure" output="Get \"https://10.134.0.16:8443/health\": dial tcp 10.134.0.16:8443: connect: connection refused" Sep 09 23:11:36.091184 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:36.091131 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls\") pod \"dns-default-tdvkg\" (UID: \"e9be36cc-dc4f-414f-98cb-ce27d9d70ff2\") " pod="openshift-dns/dns-default-tdvkg" Sep 09 23:11:36.093712 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:36.093680 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9be36cc-dc4f-414f-98cb-ce27d9d70ff2-metrics-tls\") pod \"dns-default-tdvkg\" (UID: \"e9be36cc-dc4f-414f-98cb-ce27d9d70ff2\") " pod="openshift-dns/dns-default-tdvkg" Sep 09 23:11:36.106582 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:36.106550 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d4hxc\"" Sep 09 23:11:36.115458 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:36.115427 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tdvkg" Sep 09 23:11:36.383130 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:36.383102 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tdvkg"] Sep 09 23:11:36.385459 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:11:36.385428 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9be36cc_dc4f_414f_98cb_ce27d9d70ff2.slice/crio-b11acd0e10490b42ab55a2a6c5d6a050927152cfaa792349e7eab16393293e07 WatchSource:0}: Error finding container b11acd0e10490b42ab55a2a6c5d6a050927152cfaa792349e7eab16393293e07: Status 404 returned error can't find the container with id b11acd0e10490b42ab55a2a6c5d6a050927152cfaa792349e7eab16393293e07 Sep 09 23:11:36.817180 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:36.817142 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tdvkg" event={"ID":"e9be36cc-dc4f-414f-98cb-ce27d9d70ff2","Type":"ContainerStarted","Data":"b11acd0e10490b42ab55a2a6c5d6a050927152cfaa792349e7eab16393293e07"} Sep 09 23:11:37.766907 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:37.766866 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" podUID="02b665a3-2525-4102-87eb-419606cea310" containerName="registry" containerID="cri-o://5ce237c86a10e70fc1a1b971094354695adc61710049540b69128827dd589498" gracePeriod=30 Sep 09 23:11:38.164835 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.164806 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:11:38.312325 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.312118 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-bound-sa-token\") pod \"02b665a3-2525-4102-87eb-419606cea310\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " Sep 09 23:11:38.312325 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.312177 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02b665a3-2525-4102-87eb-419606cea310-trusted-ca\") pod \"02b665a3-2525-4102-87eb-419606cea310\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " Sep 09 23:11:38.312325 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.312216 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7bxk\" (UniqueName: \"kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-kube-api-access-n7bxk\") pod \"02b665a3-2525-4102-87eb-419606cea310\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " Sep 09 23:11:38.312325 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.312253 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-registry-tls\") pod \"02b665a3-2525-4102-87eb-419606cea310\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " Sep 09 23:11:38.312325 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.312282 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02b665a3-2525-4102-87eb-419606cea310-installation-pull-secrets\") pod \"02b665a3-2525-4102-87eb-419606cea310\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " Sep 09 23:11:38.312325 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.312298 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02b665a3-2525-4102-87eb-419606cea310-registry-certificates\") pod \"02b665a3-2525-4102-87eb-419606cea310\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " Sep 09 23:11:38.312325 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.312322 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02b665a3-2525-4102-87eb-419606cea310-ca-trust-extracted\") pod \"02b665a3-2525-4102-87eb-419606cea310\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " Sep 09 23:11:38.312771 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.312352 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02b665a3-2525-4102-87eb-419606cea310-image-registry-private-configuration\") pod \"02b665a3-2525-4102-87eb-419606cea310\" (UID: \"02b665a3-2525-4102-87eb-419606cea310\") " Sep 09 23:11:38.313287 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.313217 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02b665a3-2525-4102-87eb-419606cea310-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "02b665a3-2525-4102-87eb-419606cea310" (UID: "02b665a3-2525-4102-87eb-419606cea310"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:11:38.314030 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.313992 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02b665a3-2525-4102-87eb-419606cea310-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "02b665a3-2525-4102-87eb-419606cea310" (UID: "02b665a3-2525-4102-87eb-419606cea310"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:11:38.315494 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.315466 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "02b665a3-2525-4102-87eb-419606cea310" (UID: "02b665a3-2525-4102-87eb-419606cea310"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:11:38.316067 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.315931 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02b665a3-2525-4102-87eb-419606cea310-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "02b665a3-2525-4102-87eb-419606cea310" (UID: "02b665a3-2525-4102-87eb-419606cea310"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:11:38.316067 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.316043 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-kube-api-access-n7bxk" (OuterVolumeSpecName: "kube-api-access-n7bxk") pod "02b665a3-2525-4102-87eb-419606cea310" (UID: "02b665a3-2525-4102-87eb-419606cea310"). InnerVolumeSpecName "kube-api-access-n7bxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:11:38.316593 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.316552 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "02b665a3-2525-4102-87eb-419606cea310" (UID: "02b665a3-2525-4102-87eb-419606cea310"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:11:38.317003 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.316963 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02b665a3-2525-4102-87eb-419606cea310-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "02b665a3-2525-4102-87eb-419606cea310" (UID: "02b665a3-2525-4102-87eb-419606cea310"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:11:38.324942 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.324842 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b665a3-2525-4102-87eb-419606cea310-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "02b665a3-2525-4102-87eb-419606cea310" (UID: "02b665a3-2525-4102-87eb-419606cea310"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:11:38.413516 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.413469 2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-bound-sa-token\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:11:38.413516 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.413507 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02b665a3-2525-4102-87eb-419606cea310-trusted-ca\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:11:38.413516 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.413520 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7bxk\" (UniqueName: \"kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-kube-api-access-n7bxk\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:11:38.413818 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.413530 2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02b665a3-2525-4102-87eb-419606cea310-registry-tls\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:11:38.413818 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.413539 2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02b665a3-2525-4102-87eb-419606cea310-installation-pull-secrets\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:11:38.413818 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.413546 2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02b665a3-2525-4102-87eb-419606cea310-registry-certificates\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:11:38.413818 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.413554 2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02b665a3-2525-4102-87eb-419606cea310-ca-trust-extracted\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:11:38.413818 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.413564 2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02b665a3-2525-4102-87eb-419606cea310-image-registry-private-configuration\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:11:38.556321 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.556281 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-v6wvd"] Sep 09 23:11:38.556651 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.556636 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02b665a3-2525-4102-87eb-419606cea310" containerName="registry" Sep 09 23:11:38.556707 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.556655 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="02b665a3-2525-4102-87eb-419606cea310" containerName="registry" Sep 09 23:11:38.556751 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.556735 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="02b665a3-2525-4102-87eb-419606cea310" containerName="registry" Sep 09 23:11:38.561866 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.561826 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v6wvd" Sep 09 23:11:38.564592 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.564291 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 09 23:11:38.564592 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.564338 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 09 23:11:38.564592 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.564291 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 09 23:11:38.565114 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.565079 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 09 23:11:38.565257 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.565079 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mmmtt\"" Sep 09 23:11:38.565257 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.565079 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 09 23:11:38.569434 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.569405 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-v6wvd"] Sep 09 23:11:38.572205 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.572191 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n"] Sep 09 23:11:38.576381 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.575827 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-78brr"] Sep 09 23:11:38.576381 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.575980 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n" Sep 09 23:11:38.578952 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.578287 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 09 23:11:38.578952 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.578545 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 09 23:11:38.578952 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.578760 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2fqrp\"" Sep 09 23:11:38.579781 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.579246 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 09 23:11:38.579781 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.579398 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.581999 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.581763 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 09 23:11:38.581999 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.581811 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 09 23:11:38.582165 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.582033 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 09 23:11:38.582165 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.582104 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-d8l4b\"" Sep 09 23:11:38.587962 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.587774 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n"] Sep 09 23:11:38.716581 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.716540 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eeb3e51b-bd93-4733-8149-977b136d46b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.716581 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.716595 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6929f83e-bb61-4ee4-b7e2-54785f9746cf-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-fjn7n\" (UID: \"6929f83e-bb61-4ee4-b7e2-54785f9746cf\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n" Sep 09 23:11:38.716871 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.716627 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6929f83e-bb61-4ee4-b7e2-54785f9746cf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-fjn7n\" (UID: \"6929f83e-bb61-4ee4-b7e2-54785f9746cf\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n" Sep 09 23:11:38.716871 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.716656 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6929f83e-bb61-4ee4-b7e2-54785f9746cf-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-fjn7n\" (UID: \"6929f83e-bb61-4ee4-b7e2-54785f9746cf\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n" Sep 09 23:11:38.716871 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.716694 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjpbr\" (UniqueName: \"kubernetes.io/projected/d33d9ae1-397a-4288-beea-c41cb515a274-kube-api-access-kjpbr\") pod \"openshift-state-metrics-657fd6c54-v6wvd\" (UID: \"d33d9ae1-397a-4288-beea-c41cb515a274\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v6wvd" Sep 09 23:11:38.716871 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.716722 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d33d9ae1-397a-4288-beea-c41cb515a274-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-v6wvd\" (UID: \"d33d9ae1-397a-4288-beea-c41cb515a274\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v6wvd" Sep 09 23:11:38.716871 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.716745 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eeb3e51b-bd93-4733-8149-977b136d46b0-metrics-client-ca\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.716871 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.716769 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eeb3e51b-bd93-4733-8149-977b136d46b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.716871 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.716834 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eeb3e51b-bd93-4733-8149-977b136d46b0-node-exporter-tls\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.717223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.716884 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6929f83e-bb61-4ee4-b7e2-54785f9746cf-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-fjn7n\" (UID: \"6929f83e-bb61-4ee4-b7e2-54785f9746cf\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n" Sep 09 23:11:38.717223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.716909 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl4lb\" (UniqueName: \"kubernetes.io/projected/eeb3e51b-bd93-4733-8149-977b136d46b0-kube-api-access-tl4lb\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.717223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.716931 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d33d9ae1-397a-4288-beea-c41cb515a274-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-v6wvd\" (UID: \"d33d9ae1-397a-4288-beea-c41cb515a274\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v6wvd" Sep 09 23:11:38.717223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.716951 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eeb3e51b-bd93-4733-8149-977b136d46b0-sys\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.717223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.716974 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6929f83e-bb61-4ee4-b7e2-54785f9746cf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-fjn7n\" (UID: \"6929f83e-bb61-4ee4-b7e2-54785f9746cf\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n" Sep 09 23:11:38.717223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.717023 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eeb3e51b-bd93-4733-8149-977b136d46b0-node-exporter-textfile\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.717223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.717049 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eeb3e51b-bd93-4733-8149-977b136d46b0-node-exporter-wtmp\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.717223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.717096 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjfg7\" (UniqueName: \"kubernetes.io/projected/6929f83e-bb61-4ee4-b7e2-54785f9746cf-kube-api-access-mjfg7\") pod \"kube-state-metrics-74cb57df7-fjn7n\" (UID: \"6929f83e-bb61-4ee4-b7e2-54785f9746cf\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n" Sep 09 23:11:38.717223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.717137 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d33d9ae1-397a-4288-beea-c41cb515a274-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-v6wvd\" (UID: \"d33d9ae1-397a-4288-beea-c41cb515a274\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v6wvd" Sep 09 23:11:38.717223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.717160 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eeb3e51b-bd93-4733-8149-977b136d46b0-root\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.818584 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.818475 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d33d9ae1-397a-4288-beea-c41cb515a274-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-v6wvd\" (UID: \"d33d9ae1-397a-4288-beea-c41cb515a274\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v6wvd" Sep 09 23:11:38.818584 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.818529 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eeb3e51b-bd93-4733-8149-977b136d46b0-sys\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.819091 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.818597 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eeb3e51b-bd93-4733-8149-977b136d46b0-sys\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.819091 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.818633 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6929f83e-bb61-4ee4-b7e2-54785f9746cf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-fjn7n\" (UID: \"6929f83e-bb61-4ee4-b7e2-54785f9746cf\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n" Sep 09 23:11:38.819091 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.818665 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eeb3e51b-bd93-4733-8149-977b136d46b0-node-exporter-textfile\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.819091 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.818689 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eeb3e51b-bd93-4733-8149-977b136d46b0-node-exporter-wtmp\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.819091 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.818718 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjfg7\" (UniqueName: \"kubernetes.io/projected/6929f83e-bb61-4ee4-b7e2-54785f9746cf-kube-api-access-mjfg7\") pod \"kube-state-metrics-74cb57df7-fjn7n\" (UID: \"6929f83e-bb61-4ee4-b7e2-54785f9746cf\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n" Sep 09 23:11:38.819091 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.818762 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d33d9ae1-397a-4288-beea-c41cb515a274-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-v6wvd\" (UID: \"d33d9ae1-397a-4288-beea-c41cb515a274\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v6wvd" Sep 09 23:11:38.819091 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.818784 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eeb3e51b-bd93-4733-8149-977b136d46b0-root\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.819091 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.818821 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eeb3e51b-bd93-4733-8149-977b136d46b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.819091 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.818879 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6929f83e-bb61-4ee4-b7e2-54785f9746cf-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-fjn7n\" (UID: \"6929f83e-bb61-4ee4-b7e2-54785f9746cf\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n" Sep 09 23:11:38.819091 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.818911 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6929f83e-bb61-4ee4-b7e2-54785f9746cf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-fjn7n\" (UID: \"6929f83e-bb61-4ee4-b7e2-54785f9746cf\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n" Sep 09 23:11:38.819091 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.818940 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6929f83e-bb61-4ee4-b7e2-54785f9746cf-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-fjn7n\" (UID: \"6929f83e-bb61-4ee4-b7e2-54785f9746cf\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n" Sep 09 23:11:38.819091 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.818983 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjpbr\" (UniqueName: \"kubernetes.io/projected/d33d9ae1-397a-4288-beea-c41cb515a274-kube-api-access-kjpbr\") pod \"openshift-state-metrics-657fd6c54-v6wvd\" (UID: \"d33d9ae1-397a-4288-beea-c41cb515a274\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v6wvd" Sep 09 23:11:38.819091 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.819012 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d33d9ae1-397a-4288-beea-c41cb515a274-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-v6wvd\" (UID: \"d33d9ae1-397a-4288-beea-c41cb515a274\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v6wvd" Sep 09 23:11:38.819091 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.819037 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eeb3e51b-bd93-4733-8149-977b136d46b0-metrics-client-ca\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.819091 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.819062 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eeb3e51b-bd93-4733-8149-977b136d46b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.819091 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.819088 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eeb3e51b-bd93-4733-8149-977b136d46b0-node-exporter-tls\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.819815 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.819113 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6929f83e-bb61-4ee4-b7e2-54785f9746cf-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-fjn7n\" (UID: \"6929f83e-bb61-4ee4-b7e2-54785f9746cf\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n" Sep 09 23:11:38.819815 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.819142 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl4lb\" (UniqueName: \"kubernetes.io/projected/eeb3e51b-bd93-4733-8149-977b136d46b0-kube-api-access-tl4lb\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.819815 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.819189 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eeb3e51b-bd93-4733-8149-977b136d46b0-node-exporter-textfile\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.819815 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.819266 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eeb3e51b-bd93-4733-8149-977b136d46b0-root\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.819815 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.819319 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6929f83e-bb61-4ee4-b7e2-54785f9746cf-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-fjn7n\" (UID: \"6929f83e-bb61-4ee4-b7e2-54785f9746cf\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n" Sep 09 23:11:38.819815 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.819343 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eeb3e51b-bd93-4733-8149-977b136d46b0-node-exporter-wtmp\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.819815 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:11:38.819356 2561 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 09 23:11:38.819815 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:11:38.819418 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d33d9ae1-397a-4288-beea-c41cb515a274-openshift-state-metrics-tls podName:d33d9ae1-397a-4288-beea-c41cb515a274 nodeName:}" failed. No retries permitted until 2026-09-09 23:11:39.319397753 +0000 UTC m=+288.005181137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/d33d9ae1-397a-4288-beea-c41cb515a274-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-v6wvd" (UID: "d33d9ae1-397a-4288-beea-c41cb515a274") : secret "openshift-state-metrics-tls" not found Sep 09 23:11:38.819815 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.819453 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6929f83e-bb61-4ee4-b7e2-54785f9746cf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-fjn7n\" (UID: \"6929f83e-bb61-4ee4-b7e2-54785f9746cf\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n" Sep 09 23:11:38.821079 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:11:38.819931 2561 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 09 23:11:38.821079 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:11:38.819987 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eeb3e51b-bd93-4733-8149-977b136d46b0-node-exporter-tls podName:eeb3e51b-bd93-4733-8149-977b136d46b0 nodeName:}" failed. No retries permitted until 2026-09-09 23:11:39.319969565 +0000 UTC m=+288.005752946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/eeb3e51b-bd93-4733-8149-977b136d46b0-node-exporter-tls") pod "node-exporter-78brr" (UID: "eeb3e51b-bd93-4733-8149-977b136d46b0") : secret "node-exporter-tls" not found Sep 09 23:11:38.821079 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.820004 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eeb3e51b-bd93-4733-8149-977b136d46b0-metrics-client-ca\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.821079 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.820092 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d33d9ae1-397a-4288-beea-c41cb515a274-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-v6wvd\" (UID: \"d33d9ae1-397a-4288-beea-c41cb515a274\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v6wvd" Sep 09 23:11:38.821079 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.820637 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eeb3e51b-bd93-4733-8149-977b136d46b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.821079 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.820667 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6929f83e-bb61-4ee4-b7e2-54785f9746cf-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-fjn7n\" (UID: \"6929f83e-bb61-4ee4-b7e2-54785f9746cf\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n" Sep 09 23:11:38.823442 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.822499 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d33d9ae1-397a-4288-beea-c41cb515a274-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-v6wvd\" (UID: \"d33d9ae1-397a-4288-beea-c41cb515a274\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v6wvd" Sep 09 23:11:38.823442 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.822634 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6929f83e-bb61-4ee4-b7e2-54785f9746cf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-fjn7n\" (UID: \"6929f83e-bb61-4ee4-b7e2-54785f9746cf\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n" Sep 09 23:11:38.823442 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.823004 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6929f83e-bb61-4ee4-b7e2-54785f9746cf-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-fjn7n\" (UID: \"6929f83e-bb61-4ee4-b7e2-54785f9746cf\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n" Sep 09 23:11:38.824200 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.823760 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eeb3e51b-bd93-4733-8149-977b136d46b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.827303 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.827275 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl4lb\" (UniqueName: \"kubernetes.io/projected/eeb3e51b-bd93-4733-8149-977b136d46b0-kube-api-access-tl4lb\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:38.829401 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.829368 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tdvkg" event={"ID":"e9be36cc-dc4f-414f-98cb-ce27d9d70ff2","Type":"ContainerStarted","Data":"53433ba5b327c327a9c86799ee19ec561578abb0d24001bcf0eb8a0140347213"} Sep 09 23:11:38.829514 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.829425 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tdvkg" event={"ID":"e9be36cc-dc4f-414f-98cb-ce27d9d70ff2","Type":"ContainerStarted","Data":"9994e7e0422208d19c231372a551865cf4f91af1531d9af123228b073852aa03"} Sep 09 23:11:38.829514 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.829436 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjfg7\" (UniqueName: \"kubernetes.io/projected/6929f83e-bb61-4ee4-b7e2-54785f9746cf-kube-api-access-mjfg7\") pod \"kube-state-metrics-74cb57df7-fjn7n\" (UID: \"6929f83e-bb61-4ee4-b7e2-54785f9746cf\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n" Sep 09 23:11:38.829646 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.829513 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tdvkg" Sep 09 23:11:38.829985 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.829750 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjpbr\" (UniqueName: \"kubernetes.io/projected/d33d9ae1-397a-4288-beea-c41cb515a274-kube-api-access-kjpbr\") pod \"openshift-state-metrics-657fd6c54-v6wvd\" (UID: \"d33d9ae1-397a-4288-beea-c41cb515a274\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v6wvd" Sep 09 23:11:38.831153 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.831129 2561 generic.go:356] "Generic (PLEG): container finished" podID="02b665a3-2525-4102-87eb-419606cea310" containerID="5ce237c86a10e70fc1a1b971094354695adc61710049540b69128827dd589498" exitCode=0 Sep 09 23:11:38.831244 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.831186 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" Sep 09 23:11:38.831244 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.831195 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" event={"ID":"02b665a3-2525-4102-87eb-419606cea310","Type":"ContainerDied","Data":"5ce237c86a10e70fc1a1b971094354695adc61710049540b69128827dd589498"} Sep 09 23:11:38.831244 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.831241 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56cd69fd8c-bj9b8" event={"ID":"02b665a3-2525-4102-87eb-419606cea310","Type":"ContainerDied","Data":"b3d1a68f691482aaebb07fd22c976519c1026afde885e8f7f683ffa7af33cba3"} Sep 09 23:11:38.831361 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.831269 2561 scope.go:117] "RemoveContainer" containerID="5ce237c86a10e70fc1a1b971094354695adc61710049540b69128827dd589498" Sep 09 23:11:38.840274 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.840255 2561 scope.go:117] "RemoveContainer" containerID="5ce237c86a10e70fc1a1b971094354695adc61710049540b69128827dd589498" Sep 09 23:11:38.840947 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:11:38.840839 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ce237c86a10e70fc1a1b971094354695adc61710049540b69128827dd589498\": container with ID starting with 5ce237c86a10e70fc1a1b971094354695adc61710049540b69128827dd589498 not found: ID does not exist" containerID="5ce237c86a10e70fc1a1b971094354695adc61710049540b69128827dd589498" Sep 09 23:11:38.840947 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.840908 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ce237c86a10e70fc1a1b971094354695adc61710049540b69128827dd589498"} err="failed to get container status \"5ce237c86a10e70fc1a1b971094354695adc61710049540b69128827dd589498\": rpc error: code = NotFound desc = could not find container \"5ce237c86a10e70fc1a1b971094354695adc61710049540b69128827dd589498\": container with ID starting with 5ce237c86a10e70fc1a1b971094354695adc61710049540b69128827dd589498 not found: ID does not exist" Sep 09 23:11:38.845301 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.845251 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tdvkg" podStartSLOduration=251.16761091 podStartE2EDuration="4m12.845231522s" totalImagesPullingTime="1.677620612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:07:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:11:38.844436271 +0000 UTC m=+287.530219689" watchObservedRunningTime="2026-09-09 23:11:38.845231522 +0000 UTC m=+287.531014926" Sep 09 23:11:38.860036 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.860003 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56cd69fd8c-bj9b8"] Sep 09 23:11:38.865751 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.865722 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-56cd69fd8c-bj9b8"] Sep 09 23:11:38.891869 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:38.891826 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n" Sep 09 23:11:39.168593 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:39.168567 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n"] Sep 09 23:11:39.171234 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:11:39.171205 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6929f83e_bb61_4ee4_b7e2_54785f9746cf.slice/crio-153adf1abc689b4a7c474054893d9cb4d35b1a2b2b96aa5cd4303004edd77d2c WatchSource:0}: Error finding container 153adf1abc689b4a7c474054893d9cb4d35b1a2b2b96aa5cd4303004edd77d2c: Status 404 returned error can't find the container with id 153adf1abc689b4a7c474054893d9cb4d35b1a2b2b96aa5cd4303004edd77d2c Sep 09 23:11:39.324313 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:39.324273 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eeb3e51b-bd93-4733-8149-977b136d46b0-node-exporter-tls\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:39.324472 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:39.324340 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d33d9ae1-397a-4288-beea-c41cb515a274-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-v6wvd\" (UID: \"d33d9ae1-397a-4288-beea-c41cb515a274\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v6wvd" Sep 09 23:11:39.326951 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:39.326920 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eeb3e51b-bd93-4733-8149-977b136d46b0-node-exporter-tls\") pod \"node-exporter-78brr\" (UID: \"eeb3e51b-bd93-4733-8149-977b136d46b0\") " pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:39.327077 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:39.326996 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d33d9ae1-397a-4288-beea-c41cb515a274-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-v6wvd\" (UID: \"d33d9ae1-397a-4288-beea-c41cb515a274\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v6wvd" Sep 09 23:11:39.474815 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:39.474778 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v6wvd" Sep 09 23:11:39.500160 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:39.500126 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-78brr" Sep 09 23:11:39.516559 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:11:39.516520 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb3e51b_bd93_4733_8149_977b136d46b0.slice/crio-05abb557b60df9b7416491bb671664e7d0509c5b7918e27a18230310ba55654c WatchSource:0}: Error finding container 05abb557b60df9b7416491bb671664e7d0509c5b7918e27a18230310ba55654c: Status 404 returned error can't find the container with id 05abb557b60df9b7416491bb671664e7d0509c5b7918e27a18230310ba55654c Sep 09 23:11:39.753157 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:39.753133 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-v6wvd"] Sep 09 23:11:39.755570 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:11:39.755541 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd33d9ae1_397a_4288_beea_c41cb515a274.slice/crio-49a6f014a63da40b13c39257232aebd5365431589b76db2d0af58bb3cb15bd15 WatchSource:0}: Error finding container 49a6f014a63da40b13c39257232aebd5365431589b76db2d0af58bb3cb15bd15: Status 404 returned error can't find the container with id 49a6f014a63da40b13c39257232aebd5365431589b76db2d0af58bb3cb15bd15 Sep 09 23:11:39.810304 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:39.810277 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02b665a3-2525-4102-87eb-419606cea310" path="/var/lib/kubelet/pods/02b665a3-2525-4102-87eb-419606cea310/volumes" Sep 09 23:11:39.838268 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:39.838233 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v6wvd" event={"ID":"d33d9ae1-397a-4288-beea-c41cb515a274","Type":"ContainerStarted","Data":"c9b9904119b728b9dd5225bcb79596d1252b10a137fac1a76c4acc9134660ebd"} Sep 09 23:11:39.838670 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:39.838311 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v6wvd" event={"ID":"d33d9ae1-397a-4288-beea-c41cb515a274","Type":"ContainerStarted","Data":"49a6f014a63da40b13c39257232aebd5365431589b76db2d0af58bb3cb15bd15"} Sep 09 23:11:39.839628 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:39.839606 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n" event={"ID":"6929f83e-bb61-4ee4-b7e2-54785f9746cf","Type":"ContainerStarted","Data":"153adf1abc689b4a7c474054893d9cb4d35b1a2b2b96aa5cd4303004edd77d2c"} Sep 09 23:11:39.842212 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:39.842182 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-78brr" event={"ID":"eeb3e51b-bd93-4733-8149-977b136d46b0","Type":"ContainerStarted","Data":"05abb557b60df9b7416491bb671664e7d0509c5b7918e27a18230310ba55654c"} Sep 09 23:11:40.848773 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:40.848740 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v6wvd" event={"ID":"d33d9ae1-397a-4288-beea-c41cb515a274","Type":"ContainerStarted","Data":"41f69154e0bda852586342e77b2227fbe49f5d189c52911a0ca6e423ba166d9a"} Sep 09 23:11:41.855533 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:41.855499 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v6wvd" event={"ID":"d33d9ae1-397a-4288-beea-c41cb515a274","Type":"ContainerStarted","Data":"856c7ec6135357a72165f434e958b976dd948a765ef9ad40ad7539240c01d7c7"} Sep 09 23:11:41.858169 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:41.858134 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n" event={"ID":"6929f83e-bb61-4ee4-b7e2-54785f9746cf","Type":"ContainerStarted","Data":"ab3ed2fd767b32e5e1f4b9aec55a0e90979823b94535b97bff3276ed90b62c5e"} Sep 09 23:11:41.858312 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:41.858197 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n" event={"ID":"6929f83e-bb61-4ee4-b7e2-54785f9746cf","Type":"ContainerStarted","Data":"236f68535586827a5d3552e85209e6263fd4b9c7600df7e26e64ed0d801b12c8"} Sep 09 23:11:41.858312 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:41.858228 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n" event={"ID":"6929f83e-bb61-4ee4-b7e2-54785f9746cf","Type":"ContainerStarted","Data":"e6af859a53a62f2bb6a5d6c7aa918419590469803ca6d41379a47946b0c0b1fc"} Sep 09 23:11:41.859827 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:41.859806 2561 generic.go:356] "Generic (PLEG): container finished" podID="eeb3e51b-bd93-4733-8149-977b136d46b0" containerID="f2fd1c5ecca7da82889896b3e94416ba6bc4dcfa244561e200bd2f2bd35fbcac" exitCode=0 Sep 09 23:11:41.859956 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:41.859882 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-78brr" event={"ID":"eeb3e51b-bd93-4733-8149-977b136d46b0","Type":"ContainerDied","Data":"f2fd1c5ecca7da82889896b3e94416ba6bc4dcfa244561e200bd2f2bd35fbcac"} Sep 09 23:11:41.875466 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:41.875424 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v6wvd" podStartSLOduration=2.656693343 podStartE2EDuration="3.875408364s" totalImagesPullingTime="1.218715021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:11:41.87451639 +0000 UTC m=+290.560299820" watchObservedRunningTime="2026-09-09 23:11:41.875408364 +0000 UTC m=+290.561191767" Sep 09 23:11:41.891048 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:41.890999 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fjn7n" podStartSLOduration=1.952085264 podStartE2EDuration="3.890981087s" totalImagesPullingTime="1.938895823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:11:41.889414557 +0000 UTC m=+290.575198081" watchObservedRunningTime="2026-09-09 23:11:41.890981087 +0000 UTC m=+290.576764493" Sep 09 23:11:42.579401 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.579364 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5bff948996-pb87r"] Sep 09 23:11:42.582786 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.582765 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:42.585202 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.585174 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 09 23:11:42.585320 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.585174 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 09 23:11:42.585392 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.585370 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-jg5ss\"" Sep 09 23:11:42.585451 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.585411 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 09 23:11:42.585518 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.585460 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 09 23:11:42.585518 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.585460 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8sfgr9mfha4dv\"" Sep 09 23:11:42.585964 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.585948 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 09 23:11:42.592740 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.592719 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5bff948996-pb87r"] Sep 09 23:11:42.650945 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.650904 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eb5a4583-4f79-4391-80c1-72c88355e67b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bff948996-pb87r\" (UID: \"eb5a4583-4f79-4391-80c1-72c88355e67b\") " pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:42.650945 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.650945 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/eb5a4583-4f79-4391-80c1-72c88355e67b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bff948996-pb87r\" (UID: \"eb5a4583-4f79-4391-80c1-72c88355e67b\") " pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:42.651186 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.650969 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eb5a4583-4f79-4391-80c1-72c88355e67b-secret-grpc-tls\") pod \"thanos-querier-5bff948996-pb87r\" (UID: \"eb5a4583-4f79-4391-80c1-72c88355e67b\") " pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:42.651186 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.650997 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpthr\" (UniqueName: \"kubernetes.io/projected/eb5a4583-4f79-4391-80c1-72c88355e67b-kube-api-access-gpthr\") pod \"thanos-querier-5bff948996-pb87r\" (UID: \"eb5a4583-4f79-4391-80c1-72c88355e67b\") " pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:42.651186 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.651026 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb5a4583-4f79-4391-80c1-72c88355e67b-metrics-client-ca\") pod \"thanos-querier-5bff948996-pb87r\" (UID: \"eb5a4583-4f79-4391-80c1-72c88355e67b\") " pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:42.651186 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.651066 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eb5a4583-4f79-4391-80c1-72c88355e67b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bff948996-pb87r\" (UID: \"eb5a4583-4f79-4391-80c1-72c88355e67b\") " pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:42.651186 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.651088 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/eb5a4583-4f79-4391-80c1-72c88355e67b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bff948996-pb87r\" (UID: \"eb5a4583-4f79-4391-80c1-72c88355e67b\") " pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:42.651186 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.651111 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/eb5a4583-4f79-4391-80c1-72c88355e67b-secret-thanos-querier-tls\") pod \"thanos-querier-5bff948996-pb87r\" (UID: \"eb5a4583-4f79-4391-80c1-72c88355e67b\") " pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:42.752312 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.752277 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/eb5a4583-4f79-4391-80c1-72c88355e67b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bff948996-pb87r\" (UID: \"eb5a4583-4f79-4391-80c1-72c88355e67b\") " pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:42.752458 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.752313 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eb5a4583-4f79-4391-80c1-72c88355e67b-secret-grpc-tls\") pod \"thanos-querier-5bff948996-pb87r\" (UID: \"eb5a4583-4f79-4391-80c1-72c88355e67b\") " pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:42.752458 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.752350 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpthr\" (UniqueName: \"kubernetes.io/projected/eb5a4583-4f79-4391-80c1-72c88355e67b-kube-api-access-gpthr\") pod \"thanos-querier-5bff948996-pb87r\" (UID: \"eb5a4583-4f79-4391-80c1-72c88355e67b\") " pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:42.752458 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.752371 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb5a4583-4f79-4391-80c1-72c88355e67b-metrics-client-ca\") pod \"thanos-querier-5bff948996-pb87r\" (UID: \"eb5a4583-4f79-4391-80c1-72c88355e67b\") " pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:42.752458 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.752405 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eb5a4583-4f79-4391-80c1-72c88355e67b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bff948996-pb87r\" (UID: \"eb5a4583-4f79-4391-80c1-72c88355e67b\") " pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:42.752458 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.752426 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/eb5a4583-4f79-4391-80c1-72c88355e67b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bff948996-pb87r\" (UID: \"eb5a4583-4f79-4391-80c1-72c88355e67b\") " pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:42.752458 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.752458 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/eb5a4583-4f79-4391-80c1-72c88355e67b-secret-thanos-querier-tls\") pod \"thanos-querier-5bff948996-pb87r\" (UID: \"eb5a4583-4f79-4391-80c1-72c88355e67b\") " pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:42.752734 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.752515 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eb5a4583-4f79-4391-80c1-72c88355e67b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bff948996-pb87r\" (UID: \"eb5a4583-4f79-4391-80c1-72c88355e67b\") " pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:42.753448 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.753420 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb5a4583-4f79-4391-80c1-72c88355e67b-metrics-client-ca\") pod \"thanos-querier-5bff948996-pb87r\" (UID: \"eb5a4583-4f79-4391-80c1-72c88355e67b\") " pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:42.755209 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.755186 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eb5a4583-4f79-4391-80c1-72c88355e67b-secret-grpc-tls\") pod \"thanos-querier-5bff948996-pb87r\" (UID: \"eb5a4583-4f79-4391-80c1-72c88355e67b\") " pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:42.755735 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.755713 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/eb5a4583-4f79-4391-80c1-72c88355e67b-secret-thanos-querier-tls\") pod \"thanos-querier-5bff948996-pb87r\" (UID: \"eb5a4583-4f79-4391-80c1-72c88355e67b\") " pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:42.755841 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.755762 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eb5a4583-4f79-4391-80c1-72c88355e67b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bff948996-pb87r\" (UID: \"eb5a4583-4f79-4391-80c1-72c88355e67b\") " pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:42.755841 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.755763 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/eb5a4583-4f79-4391-80c1-72c88355e67b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bff948996-pb87r\" (UID: \"eb5a4583-4f79-4391-80c1-72c88355e67b\") " pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:42.755841 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.755776 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eb5a4583-4f79-4391-80c1-72c88355e67b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bff948996-pb87r\" (UID: \"eb5a4583-4f79-4391-80c1-72c88355e67b\") " pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:42.756005 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.755987 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/eb5a4583-4f79-4391-80c1-72c88355e67b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bff948996-pb87r\" (UID: \"eb5a4583-4f79-4391-80c1-72c88355e67b\") " pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:42.759764 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.759746 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpthr\" (UniqueName: \"kubernetes.io/projected/eb5a4583-4f79-4391-80c1-72c88355e67b-kube-api-access-gpthr\") pod \"thanos-querier-5bff948996-pb87r\" (UID: \"eb5a4583-4f79-4391-80c1-72c88355e67b\") " pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:42.865708 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.865605 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-78brr" event={"ID":"eeb3e51b-bd93-4733-8149-977b136d46b0","Type":"ContainerStarted","Data":"b311ac05a8c2e33bcc0d9f1e3b21575a8191cdc16e30500c0f8e2a7115e0ea2c"} Sep 09 23:11:42.865708 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.865667 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-78brr" event={"ID":"eeb3e51b-bd93-4733-8149-977b136d46b0","Type":"ContainerStarted","Data":"52ec18823355da71316ea712c0e1e559486462f894d64bb4725358652663dcb2"} Sep 09 23:11:42.886409 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.886353 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-78brr" podStartSLOduration=3.2065422789999998 podStartE2EDuration="4.886334138s" totalImagesPullingTime="1.593720437s" totalInitContainerRuntime="86.071422ms" isStatefulPod=true podCreationTimestamp="2026-09-09 23:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:11:42.885252326 +0000 UTC m=+291.571035778" watchObservedRunningTime="2026-09-09 23:11:42.886334138 +0000 UTC m=+291.572117541" Sep 09 23:11:42.892722 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:42.892685 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:43.174112 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:43.174085 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5bff948996-pb87r"] Sep 09 23:11:43.176155 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:11:43.176126 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb5a4583_4f79_4391_80c1_72c88355e67b.slice/crio-c9f08871cfe16666eb44c83dc343737de1ba8add64060ad09faa0bd101bfe932 WatchSource:0}: Error finding container c9f08871cfe16666eb44c83dc343737de1ba8add64060ad09faa0bd101bfe932: Status 404 returned error can't find the container with id c9f08871cfe16666eb44c83dc343737de1ba8add64060ad09faa0bd101bfe932 Sep 09 23:11:43.374878 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:43.374015 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fd4b45f9b-lgxgd"] Sep 09 23:11:43.871800 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:43.871764 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" event={"ID":"eb5a4583-4f79-4391-80c1-72c88355e67b","Type":"ContainerStarted","Data":"c9f08871cfe16666eb44c83dc343737de1ba8add64060ad09faa0bd101bfe932"} Sep 09 23:11:44.782223 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.782181 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 23:11:44.786473 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.786445 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.788892 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.788838 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 09 23:11:44.789035 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.788909 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2cl8d\"" Sep 09 23:11:44.789035 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.788949 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 09 23:11:44.789141 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.789127 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 09 23:11:44.789201 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.789177 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 09 23:11:44.789338 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.789313 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 09 23:11:44.789876 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.789827 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 09 23:11:44.789966 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.789910 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 09 23:11:44.790200 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.790176 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bf7v5dqr0h4u4\"" Sep 09 23:11:44.791480 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.791005 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 09 23:11:44.791480 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.791094 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 09 23:11:44.791480 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.791204 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 09 23:11:44.791480 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.791223 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 09 23:11:44.791480 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.791096 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 09 23:11:44.791795 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.791525 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 09 23:11:44.793637 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.793618 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 09 23:11:44.796502 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.796480 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 09 23:11:44.804641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.804618 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 23:11:44.873247 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.873213 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.873247 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.873253 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.873753 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.873277 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.873753 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.873308 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.873753 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.873373 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-config\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.873753 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.873436 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.873753 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.873474 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.873753 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.873513 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-655tc\" (UniqueName: \"kubernetes.io/projected/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-kube-api-access-655tc\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.873753 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.873554 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.873753 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.873587 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.873753 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.873627 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.873753 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.873652 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.873753 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.873674 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-config-out\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.873753 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.873748 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.874187 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.873771 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-web-config\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.874187 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.873804 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.874187 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.873818 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.874187 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.873835 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.874187 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.873871 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.874187 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.873924 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.975079 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.975043 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.975079 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.975084 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.975345 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.975117 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.975345 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.975282 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.975345 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.975337 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.975485 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.975369 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.975485 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.975397 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.975485 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.975428 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.975485 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.975459 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.975692 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.975489 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-config\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.975692 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.975520 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.975692 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.975543 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.975692 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.975571 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-655tc\" (UniqueName: \"kubernetes.io/projected/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-kube-api-access-655tc\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.975692 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.975576 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.975692 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.975601 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.975692 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.975631 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.975692 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.975665 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.975692 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.975693 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.976296 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.975720 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-config-out\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.976296 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.975790 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.976296 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.975813 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-web-config\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.977167 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.977107 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.977167 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.977127 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.980039 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.980012 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.981143 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.980472 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.981143 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.980760 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.981143 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.980894 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.981143 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.980925 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.981843 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.981798 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.982004 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.981980 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.982507 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.982262 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.983563 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.983093 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.983563 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.983480 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-config\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.983563 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.983521 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.983828 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.983787 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-config-out\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.983984 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.983901 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.984195 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.984172 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.984797 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.984775 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.985214 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.985172 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-web-config\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:44.985292 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:44.985270 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-655tc\" (UniqueName: \"kubernetes.io/projected/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-kube-api-access-655tc\") pod \"prometheus-k8s-0\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:45.101390 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:45.101301 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:11:45.484022 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:45.483992 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 23:11:45.485738 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:11:45.485698 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba0feeb_146f_4bdf_997d_3ea9d54b25e4.slice/crio-4923b52963144d1dcc12c24f8c3b8a49763802dc0b8ce664224c4494262d3ada WatchSource:0}: Error finding container 4923b52963144d1dcc12c24f8c3b8a49763802dc0b8ce664224c4494262d3ada: Status 404 returned error can't find the container with id 4923b52963144d1dcc12c24f8c3b8a49763802dc0b8ce664224c4494262d3ada Sep 09 23:11:45.881987 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:45.881949 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" event={"ID":"eb5a4583-4f79-4391-80c1-72c88355e67b","Type":"ContainerStarted","Data":"b1ba50a8e3cf6a3b0fc2c166a2da6d439ead24009952c9e908a9cf194c6f0b9b"} Sep 09 23:11:45.882394 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:45.882032 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" event={"ID":"eb5a4583-4f79-4391-80c1-72c88355e67b","Type":"ContainerStarted","Data":"d4f335e5ecff3fd96fcd17676a116825b17c2973941e7ae65f5cd158582cfaac"} Sep 09 23:11:45.882394 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:45.882070 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" event={"ID":"eb5a4583-4f79-4391-80c1-72c88355e67b","Type":"ContainerStarted","Data":"3543892a610c49959a37fd95cd747fecccb595abeaa70e2f309b8d825d5ae877"} Sep 09 23:11:45.883146 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:45.883128 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4","Type":"ContainerStarted","Data":"4923b52963144d1dcc12c24f8c3b8a49763802dc0b8ce664224c4494262d3ada"} Sep 09 23:11:47.899563 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:47.899524 2561 generic.go:356] "Generic (PLEG): container finished" podID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerID="de99b2098e53175b582ac267954b35df4dd975edf9baec94cf967adb4514ef18" exitCode=0 Sep 09 23:11:47.899980 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:47.899599 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4","Type":"ContainerDied","Data":"de99b2098e53175b582ac267954b35df4dd975edf9baec94cf967adb4514ef18"} Sep 09 23:11:47.903074 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:47.903051 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" event={"ID":"eb5a4583-4f79-4391-80c1-72c88355e67b","Type":"ContainerStarted","Data":"b23353be5aa5b57c9209cbe5397dd4fa574c6940e713d2c896700162d041b314"} Sep 09 23:11:47.903172 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:47.903127 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" event={"ID":"eb5a4583-4f79-4391-80c1-72c88355e67b","Type":"ContainerStarted","Data":"ff992ae846045ddc4f263ad4043cfeac66b81f72806319d331375dbb2282c553"} Sep 09 23:11:47.903230 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:47.903170 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" event={"ID":"eb5a4583-4f79-4391-80c1-72c88355e67b","Type":"ContainerStarted","Data":"cf531ea94a3eb1e8f890a3b8c5976ced53acf1c8be3812fb99083ba6ab61eaa8"} Sep 09 23:11:47.903276 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:47.903234 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:47.943079 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:47.943025 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" podStartSLOduration=2.506794115 podStartE2EDuration="5.94300574s" totalImagesPullingTime="3.436211625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:11:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:11:47.941495653 +0000 UTC m=+296.627279071" watchObservedRunningTime="2026-09-09 23:11:47.94300574 +0000 UTC m=+296.628789190" Sep 09 23:11:48.845603 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:48.845569 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tdvkg" Sep 09 23:11:51.693207 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:51.693177 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2fs7f_0a127828-4a97-40cf-b8ae-f5ecd20f97c6/console-operator/1.log" Sep 09 23:11:51.694062 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:51.694039 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2fs7f_0a127828-4a97-40cf-b8ae-f5ecd20f97c6/console-operator/1.log" Sep 09 23:11:51.711230 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:51.711207 2561 kubelet.go:1648] "Image garbage collection succeeded" Sep 09 23:11:51.924941 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:51.924766 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4","Type":"ContainerStarted","Data":"a65620d50494872011a8423ba15a40171234d165befcdb7e4d7723693404dc1e"} Sep 09 23:11:51.924941 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:51.924840 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4","Type":"ContainerStarted","Data":"85c15df8ecfc089f6b66a0bcda378b50167962bdb4bd729564b34e49000cc465"} Sep 09 23:11:51.924941 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:51.924885 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4","Type":"ContainerStarted","Data":"a0bf9ac6785b1361017ab10844d57a6ba3bda1386c15d7cf596e6354456bee0f"} Sep 09 23:11:51.924941 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:51.924909 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4","Type":"ContainerStarted","Data":"9f3f533b1dc98b911c006d37ddece210df426b3be61dfcdef09d1c37fe363077"} Sep 09 23:11:51.924941 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:51.924935 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4","Type":"ContainerStarted","Data":"ba914d835a778e595a52a377a4ec689deb2ae9c7d3a379a3145ffa57ce79cac1"} Sep 09 23:11:51.959054 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:51.924958 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4","Type":"ContainerStarted","Data":"e6e6d319348949c4150f71a96a095b53f6c4c71dccd162201bab15f473b47039"} Sep 09 23:11:51.959054 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:51.954515 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.662963147 podStartE2EDuration="7.954499312s" totalImagesPullingTime="4.265773965s" totalInitContainerRuntime="25.7622ms" isStatefulPod=false podCreationTimestamp="2026-09-09 23:11:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:11:51.953692782 +0000 UTC m=+300.639476187" watchObservedRunningTime="2026-09-09 23:11:51.954499312 +0000 UTC m=+300.640282715" Sep 09 23:11:53.914486 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:53.914455 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5bff948996-pb87r" Sep 09 23:11:55.102470 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:11:55.102419 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:12:08.397176 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.397113 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5fd4b45f9b-lgxgd" podUID="612bca39-7119-4574-ad6b-16ae0adc95e4" containerName="console" containerID="cri-o://6c44fa49728029a607f4986c5832966b3b02c6b6f3b62c36c15dd6e75c1bf433" gracePeriod=15 Sep 09 23:12:08.634666 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.634634 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:12:08.683841 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.683812 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/612bca39-7119-4574-ad6b-16ae0adc95e4-trusted-ca-bundle\") pod \"612bca39-7119-4574-ad6b-16ae0adc95e4\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " Sep 09 23:12:08.683991 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.683886 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/612bca39-7119-4574-ad6b-16ae0adc95e4-oauth-serving-cert\") pod \"612bca39-7119-4574-ad6b-16ae0adc95e4\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " Sep 09 23:12:08.683991 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.683917 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrfm4\" (UniqueName: \"kubernetes.io/projected/612bca39-7119-4574-ad6b-16ae0adc95e4-kube-api-access-qrfm4\") pod \"612bca39-7119-4574-ad6b-16ae0adc95e4\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " Sep 09 23:12:08.683991 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.683975 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/612bca39-7119-4574-ad6b-16ae0adc95e4-service-ca\") pod \"612bca39-7119-4574-ad6b-16ae0adc95e4\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " Sep 09 23:12:08.684134 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.683998 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/612bca39-7119-4574-ad6b-16ae0adc95e4-console-serving-cert\") pod \"612bca39-7119-4574-ad6b-16ae0adc95e4\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " Sep 09 23:12:08.684134 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.684042 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/612bca39-7119-4574-ad6b-16ae0adc95e4-console-oauth-config\") pod \"612bca39-7119-4574-ad6b-16ae0adc95e4\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " Sep 09 23:12:08.684134 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.684095 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/612bca39-7119-4574-ad6b-16ae0adc95e4-console-config\") pod \"612bca39-7119-4574-ad6b-16ae0adc95e4\" (UID: \"612bca39-7119-4574-ad6b-16ae0adc95e4\") " Sep 09 23:12:08.684419 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.684331 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/612bca39-7119-4574-ad6b-16ae0adc95e4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "612bca39-7119-4574-ad6b-16ae0adc95e4" (UID: "612bca39-7119-4574-ad6b-16ae0adc95e4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:12:08.684552 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.684428 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/612bca39-7119-4574-ad6b-16ae0adc95e4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "612bca39-7119-4574-ad6b-16ae0adc95e4" (UID: "612bca39-7119-4574-ad6b-16ae0adc95e4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:12:08.684626 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.684549 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/612bca39-7119-4574-ad6b-16ae0adc95e4-service-ca" (OuterVolumeSpecName: "service-ca") pod "612bca39-7119-4574-ad6b-16ae0adc95e4" (UID: "612bca39-7119-4574-ad6b-16ae0adc95e4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:12:08.684675 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.684627 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/612bca39-7119-4574-ad6b-16ae0adc95e4-console-config" (OuterVolumeSpecName: "console-config") pod "612bca39-7119-4574-ad6b-16ae0adc95e4" (UID: "612bca39-7119-4574-ad6b-16ae0adc95e4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:12:08.686519 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.686493 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612bca39-7119-4574-ad6b-16ae0adc95e4-kube-api-access-qrfm4" (OuterVolumeSpecName: "kube-api-access-qrfm4") pod "612bca39-7119-4574-ad6b-16ae0adc95e4" (UID: "612bca39-7119-4574-ad6b-16ae0adc95e4"). InnerVolumeSpecName "kube-api-access-qrfm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:12:08.686610 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.686540 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/612bca39-7119-4574-ad6b-16ae0adc95e4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "612bca39-7119-4574-ad6b-16ae0adc95e4" (UID: "612bca39-7119-4574-ad6b-16ae0adc95e4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:12:08.686610 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.686552 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/612bca39-7119-4574-ad6b-16ae0adc95e4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "612bca39-7119-4574-ad6b-16ae0adc95e4" (UID: "612bca39-7119-4574-ad6b-16ae0adc95e4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:12:08.784865 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.784824 2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/612bca39-7119-4574-ad6b-16ae0adc95e4-console-oauth-config\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:12:08.785041 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.784882 2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/612bca39-7119-4574-ad6b-16ae0adc95e4-console-config\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:12:08.785041 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.784892 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/612bca39-7119-4574-ad6b-16ae0adc95e4-trusted-ca-bundle\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:12:08.785041 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.784900 2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/612bca39-7119-4574-ad6b-16ae0adc95e4-oauth-serving-cert\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:12:08.785041 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.784908 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrfm4\" (UniqueName: \"kubernetes.io/projected/612bca39-7119-4574-ad6b-16ae0adc95e4-kube-api-access-qrfm4\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:12:08.785041 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.784917 2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/612bca39-7119-4574-ad6b-16ae0adc95e4-service-ca\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:12:08.785041 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.784925 2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/612bca39-7119-4574-ad6b-16ae0adc95e4-console-serving-cert\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:12:08.997096 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.997004 2561 generic.go:356] "Generic (PLEG): container finished" podID="612bca39-7119-4574-ad6b-16ae0adc95e4" containerID="6c44fa49728029a607f4986c5832966b3b02c6b6f3b62c36c15dd6e75c1bf433" exitCode=0 Sep 09 23:12:08.997096 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.997052 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fd4b45f9b-lgxgd" Sep 09 23:12:08.997333 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.997097 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fd4b45f9b-lgxgd" event={"ID":"612bca39-7119-4574-ad6b-16ae0adc95e4","Type":"ContainerDied","Data":"6c44fa49728029a607f4986c5832966b3b02c6b6f3b62c36c15dd6e75c1bf433"} Sep 09 23:12:08.997333 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.997159 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fd4b45f9b-lgxgd" event={"ID":"612bca39-7119-4574-ad6b-16ae0adc95e4","Type":"ContainerDied","Data":"4c546fde3621b1858c168667eece3b8ac6f044b335726c1c1439ae2590b4d9d2"} Sep 09 23:12:08.997333 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:08.997190 2561 scope.go:117] "RemoveContainer" containerID="6c44fa49728029a607f4986c5832966b3b02c6b6f3b62c36c15dd6e75c1bf433" Sep 09 23:12:09.006133 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:09.006113 2561 scope.go:117] "RemoveContainer" containerID="6c44fa49728029a607f4986c5832966b3b02c6b6f3b62c36c15dd6e75c1bf433" Sep 09 23:12:09.006436 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:12:09.006418 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c44fa49728029a607f4986c5832966b3b02c6b6f3b62c36c15dd6e75c1bf433\": container with ID starting with 6c44fa49728029a607f4986c5832966b3b02c6b6f3b62c36c15dd6e75c1bf433 not found: ID does not exist" containerID="6c44fa49728029a607f4986c5832966b3b02c6b6f3b62c36c15dd6e75c1bf433" Sep 09 23:12:09.006498 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:09.006448 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c44fa49728029a607f4986c5832966b3b02c6b6f3b62c36c15dd6e75c1bf433"} err="failed to get container status \"6c44fa49728029a607f4986c5832966b3b02c6b6f3b62c36c15dd6e75c1bf433\": rpc error: code = NotFound desc = could not find container \"6c44fa49728029a607f4986c5832966b3b02c6b6f3b62c36c15dd6e75c1bf433\": container with ID starting with 6c44fa49728029a607f4986c5832966b3b02c6b6f3b62c36c15dd6e75c1bf433 not found: ID does not exist" Sep 09 23:12:09.016106 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:09.016068 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fd4b45f9b-lgxgd"] Sep 09 23:12:09.022612 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:09.022586 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5fd4b45f9b-lgxgd"] Sep 09 23:12:09.813576 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:09.813537 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="612bca39-7119-4574-ad6b-16ae0adc95e4" path="/var/lib/kubelet/pods/612bca39-7119-4574-ad6b-16ae0adc95e4/volumes" Sep 09 23:12:20.043641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:20.043604 2561 generic.go:356] "Generic (PLEG): container finished" podID="122b24bb-be61-4aa7-8faf-38f4c1f395f3" containerID="150b6456faad98ff01dc7736c8f8ffd3f2c7b5741a86f27cf7311bf6965e41c6" exitCode=0 Sep 09 23:12:20.044105 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:20.043655 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jk5k" event={"ID":"122b24bb-be61-4aa7-8faf-38f4c1f395f3","Type":"ContainerDied","Data":"150b6456faad98ff01dc7736c8f8ffd3f2c7b5741a86f27cf7311bf6965e41c6"} Sep 09 23:12:20.044105 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:20.043978 2561 scope.go:117] "RemoveContainer" containerID="150b6456faad98ff01dc7736c8f8ffd3f2c7b5741a86f27cf7311bf6965e41c6" Sep 09 23:12:21.050722 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:21.050688 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-8jk5k" event={"ID":"122b24bb-be61-4aa7-8faf-38f4c1f395f3","Type":"ContainerStarted","Data":"addd5932fc67c8a75f10baa420cfc7ffea654ddfb0d52675f74c4594cd9c29ac"} Sep 09 23:12:45.102132 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:45.102040 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:12:45.118985 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:45.118954 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:12:45.172649 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:12:45.172616 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:03.108599 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.108566 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 23:13:03.109178 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.108927 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="thanos-sidecar" containerID="cri-o://9f3f533b1dc98b911c006d37ddece210df426b3be61dfcdef09d1c37fe363077" gracePeriod=600 Sep 09 23:13:03.109178 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.108979 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="prometheus" containerID="cri-o://e6e6d319348949c4150f71a96a095b53f6c4c71dccd162201bab15f473b47039" gracePeriod=600 Sep 09 23:13:03.109178 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.108961 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a65620d50494872011a8423ba15a40171234d165befcdb7e4d7723693404dc1e" gracePeriod=600 Sep 09 23:13:03.109178 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.108935 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="config-reloader" containerID="cri-o://ba914d835a778e595a52a377a4ec689deb2ae9c7d3a379a3145ffa57ce79cac1" gracePeriod=600 Sep 09 23:13:03.109178 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.108967 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="kube-rbac-proxy-web" containerID="cri-o://a0bf9ac6785b1361017ab10844d57a6ba3bda1386c15d7cf596e6354456bee0f" gracePeriod=600 Sep 09 23:13:03.109178 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.108925 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="kube-rbac-proxy" containerID="cri-o://85c15df8ecfc089f6b66a0bcda378b50167962bdb4bd729564b34e49000cc465" gracePeriod=600 Sep 09 23:13:03.230490 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.230462 2561 generic.go:356] "Generic (PLEG): container finished" podID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerID="a65620d50494872011a8423ba15a40171234d165befcdb7e4d7723693404dc1e" exitCode=0 Sep 09 23:13:03.230490 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.230486 2561 generic.go:356] "Generic (PLEG): container finished" podID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerID="85c15df8ecfc089f6b66a0bcda378b50167962bdb4bd729564b34e49000cc465" exitCode=0 Sep 09 23:13:03.230490 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.230492 2561 generic.go:356] "Generic (PLEG): container finished" podID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerID="a0bf9ac6785b1361017ab10844d57a6ba3bda1386c15d7cf596e6354456bee0f" exitCode=0 Sep 09 23:13:03.230656 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.230498 2561 generic.go:356] "Generic (PLEG): container finished" podID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerID="9f3f533b1dc98b911c006d37ddece210df426b3be61dfcdef09d1c37fe363077" exitCode=0 Sep 09 23:13:03.230656 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.230504 2561 generic.go:356] "Generic (PLEG): container finished" podID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerID="ba914d835a778e595a52a377a4ec689deb2ae9c7d3a379a3145ffa57ce79cac1" exitCode=0 Sep 09 23:13:03.230656 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.230509 2561 generic.go:356] "Generic (PLEG): container finished" podID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerID="e6e6d319348949c4150f71a96a095b53f6c4c71dccd162201bab15f473b47039" exitCode=0 Sep 09 23:13:03.230656 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.230529 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4","Type":"ContainerDied","Data":"a65620d50494872011a8423ba15a40171234d165befcdb7e4d7723693404dc1e"} Sep 09 23:13:03.230656 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.230596 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4","Type":"ContainerDied","Data":"85c15df8ecfc089f6b66a0bcda378b50167962bdb4bd729564b34e49000cc465"} Sep 09 23:13:03.230656 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.230625 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4","Type":"ContainerDied","Data":"a0bf9ac6785b1361017ab10844d57a6ba3bda1386c15d7cf596e6354456bee0f"} Sep 09 23:13:03.230656 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.230647 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4","Type":"ContainerDied","Data":"9f3f533b1dc98b911c006d37ddece210df426b3be61dfcdef09d1c37fe363077"} Sep 09 23:13:03.230877 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.230670 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4","Type":"ContainerDied","Data":"ba914d835a778e595a52a377a4ec689deb2ae9c7d3a379a3145ffa57ce79cac1"} Sep 09 23:13:03.230877 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.230694 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4","Type":"ContainerDied","Data":"e6e6d319348949c4150f71a96a095b53f6c4c71dccd162201bab15f473b47039"} Sep 09 23:13:03.352066 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.352040 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:03.448765 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.448733 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-prometheus-k8s-tls\") pod \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " Sep 09 23:13:03.448971 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.448773 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-metrics-client-certs\") pod \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " Sep 09 23:13:03.448971 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.448792 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-655tc\" (UniqueName: \"kubernetes.io/projected/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-kube-api-access-655tc\") pod \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " Sep 09 23:13:03.448971 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.448821 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-configmap-metrics-client-ca\") pod \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " Sep 09 23:13:03.448971 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.448839 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-config\") pod \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " Sep 09 23:13:03.448971 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.448873 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-k8s-rulefiles-0\") pod \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " Sep 09 23:13:03.448971 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.448903 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-k8s-rulefiles-1\") pod \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " Sep 09 23:13:03.448971 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.448944 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-kube-rbac-proxy\") pod \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " Sep 09 23:13:03.448971 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.448975 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-k8s-db\") pod \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " Sep 09 23:13:03.449446 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.449007 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-configmap-serving-certs-ca-bundle\") pod \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " Sep 09 23:13:03.449446 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.449038 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-grpc-tls\") pod \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " Sep 09 23:13:03.449446 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.449074 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " Sep 09 23:13:03.449446 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.449099 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-thanos-prometheus-http-client-file\") pod \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " Sep 09 23:13:03.449446 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.449146 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " Sep 09 23:13:03.449446 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.449180 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-trusted-ca-bundle\") pod \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " Sep 09 23:13:03.449446 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.449202 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" (UID: "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:13:03.449446 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.449256 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-web-config\") pod \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " Sep 09 23:13:03.449446 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.449282 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-tls-assets\") pod \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " Sep 09 23:13:03.449446 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.449310 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-configmap-kubelet-serving-ca-bundle\") pod \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " Sep 09 23:13:03.449446 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.449336 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-k8s-rulefiles-2\") pod \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " Sep 09 23:13:03.449446 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.449368 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-config-out\") pod \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\" (UID: \"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4\") " Sep 09 23:13:03.450071 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.449516 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" (UID: "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:13:03.450071 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.449758 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" (UID: "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:13:03.450199 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.450170 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" (UID: "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:13:03.450259 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.450199 2561 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-configmap-metrics-client-ca\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:13:03.450259 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.450221 2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:13:03.450259 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.450241 2561 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:13:03.450779 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.450554 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" (UID: "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:13:03.451876 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.451569 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" (UID: "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:13:03.451987 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.451904 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" (UID: "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:13:03.452052 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.452002 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-kube-api-access-655tc" (OuterVolumeSpecName: "kube-api-access-655tc") pod "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" (UID: "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4"). InnerVolumeSpecName "kube-api-access-655tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:13:03.452156 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.452130 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-config" (OuterVolumeSpecName: "config") pod "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" (UID: "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:13:03.452492 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.452314 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" (UID: "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:13:03.452492 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.452419 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" (UID: "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:13:03.452662 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.452497 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" (UID: "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:13:03.452662 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.452596 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-config-out" (OuterVolumeSpecName: "config-out") pod "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" (UID: "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:13:03.452777 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.452670 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" (UID: "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:13:03.453957 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.453921 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" (UID: "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:13:03.454630 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.454601 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" (UID: "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:13:03.454928 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.454902 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" (UID: "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:13:03.455012 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.454946 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" (UID: "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:13:03.455012 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.454999 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" (UID: "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:13:03.465145 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.465119 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-web-config" (OuterVolumeSpecName: "web-config") pod "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" (UID: "8ba0feeb-146f-4bdf-997d-3ea9d54b25e4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:13:03.551444 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.551404 2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:13:03.551444 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.551435 2561 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-metrics-client-certs\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:13:03.551444 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.551445 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-655tc\" (UniqueName: \"kubernetes.io/projected/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-kube-api-access-655tc\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:13:03.551444 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.551456 2561 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-config\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:13:03.551702 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.551464 2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:13:03.551702 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.551473 2561 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-kube-rbac-proxy\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:13:03.551702 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.551481 2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-k8s-db\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:13:03.551702 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.551489 2561 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-grpc-tls\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:13:03.551702 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.551498 2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:13:03.551702 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.551507 2561 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:13:03.551702 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.551515 2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:13:03.551702 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.551524 2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:13:03.551702 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.551534 2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-web-config\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:13:03.551702 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.551542 2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-tls-assets\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:13:03.551702 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.551549 2561 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:13:03.551702 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.551557 2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:13:03.551702 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:03.551564 2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4-config-out\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:13:04.238345 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.238310 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8ba0feeb-146f-4bdf-997d-3ea9d54b25e4","Type":"ContainerDied","Data":"4923b52963144d1dcc12c24f8c3b8a49763802dc0b8ce664224c4494262d3ada"} Sep 09 23:13:04.238782 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.238362 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.238782 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.238392 2561 scope.go:117] "RemoveContainer" containerID="a65620d50494872011a8423ba15a40171234d165befcdb7e4d7723693404dc1e" Sep 09 23:13:04.246777 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.246752 2561 scope.go:117] "RemoveContainer" containerID="85c15df8ecfc089f6b66a0bcda378b50167962bdb4bd729564b34e49000cc465" Sep 09 23:13:04.254499 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.254477 2561 scope.go:117] "RemoveContainer" containerID="a0bf9ac6785b1361017ab10844d57a6ba3bda1386c15d7cf596e6354456bee0f" Sep 09 23:13:04.259816 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.259791 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 23:13:04.263023 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.262993 2561 scope.go:117] "RemoveContainer" containerID="9f3f533b1dc98b911c006d37ddece210df426b3be61dfcdef09d1c37fe363077" Sep 09 23:13:04.265308 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.265251 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 23:13:04.271995 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.271974 2561 scope.go:117] "RemoveContainer" containerID="ba914d835a778e595a52a377a4ec689deb2ae9c7d3a379a3145ffa57ce79cac1" Sep 09 23:13:04.279927 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.279908 2561 scope.go:117] "RemoveContainer" containerID="e6e6d319348949c4150f71a96a095b53f6c4c71dccd162201bab15f473b47039" Sep 09 23:13:04.287267 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.287237 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 23:13:04.287684 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.287665 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="kube-rbac-proxy-web" Sep 09 23:13:04.287731 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.287691 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="kube-rbac-proxy-web" Sep 09 23:13:04.287731 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.287714 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="kube-rbac-proxy-thanos" Sep 09 23:13:04.287731 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.287724 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="kube-rbac-proxy-thanos" Sep 09 23:13:04.287865 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.287748 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="config-reloader" Sep 09 23:13:04.287865 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.287757 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="config-reloader" Sep 09 23:13:04.287865 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.287774 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="thanos-sidecar" Sep 09 23:13:04.287865 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.287783 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="thanos-sidecar" Sep 09 23:13:04.287865 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.287793 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="kube-rbac-proxy" Sep 09 23:13:04.287865 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.287802 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="kube-rbac-proxy" Sep 09 23:13:04.287865 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.287816 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="612bca39-7119-4574-ad6b-16ae0adc95e4" containerName="console" Sep 09 23:13:04.287865 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.287824 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="612bca39-7119-4574-ad6b-16ae0adc95e4" containerName="console" Sep 09 23:13:04.287865 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.287836 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="init-config-reloader" Sep 09 23:13:04.288121 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.287845 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="init-config-reloader" Sep 09 23:13:04.288121 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.287901 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="prometheus" Sep 09 23:13:04.288121 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.287909 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="prometheus" Sep 09 23:13:04.288121 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.287987 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="thanos-sidecar" Sep 09 23:13:04.288121 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.287999 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="612bca39-7119-4574-ad6b-16ae0adc95e4" containerName="console" Sep 09 23:13:04.288121 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.288009 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="kube-rbac-proxy-web" Sep 09 23:13:04.288121 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.288018 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="config-reloader" Sep 09 23:13:04.288121 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.288027 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="kube-rbac-proxy" Sep 09 23:13:04.288121 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.288037 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="prometheus" Sep 09 23:13:04.288121 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.288046 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" containerName="kube-rbac-proxy-thanos" Sep 09 23:13:04.289218 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.289197 2561 scope.go:117] "RemoveContainer" containerID="de99b2098e53175b582ac267954b35df4dd975edf9baec94cf967adb4514ef18" Sep 09 23:13:04.293676 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.293651 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.296137 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.295948 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 09 23:13:04.296137 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.296001 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 09 23:13:04.296137 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.296022 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 09 23:13:04.296137 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.296051 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 09 23:13:04.296421 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.296266 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 09 23:13:04.296421 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.296284 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 09 23:13:04.296649 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.296629 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 09 23:13:04.296750 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.296679 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 09 23:13:04.296750 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.296698 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bf7v5dqr0h4u4\"" Sep 09 23:13:04.297107 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.296996 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 09 23:13:04.297107 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.297054 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 09 23:13:04.297271 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.297255 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 09 23:13:04.297465 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.297449 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 09 23:13:04.297806 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.297785 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2cl8d\"" Sep 09 23:13:04.298010 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.297989 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 09 23:13:04.305020 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.304978 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 09 23:13:04.307317 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.307274 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 09 23:13:04.308952 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.308918 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 23:13:04.359534 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.359491 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-config\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.359534 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.359535 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda93acf-5e54-430b-8187-b2d6265ff0d0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.359754 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.359554 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cda93acf-5e54-430b-8187-b2d6265ff0d0-config-out\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.359754 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.359614 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cda93acf-5e54-430b-8187-b2d6265ff0d0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.359754 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.359664 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2qpx\" (UniqueName: \"kubernetes.io/projected/cda93acf-5e54-430b-8187-b2d6265ff0d0-kube-api-access-z2qpx\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.359754 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.359690 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.359754 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.359714 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.359754 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.359735 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda93acf-5e54-430b-8187-b2d6265ff0d0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.360042 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.359769 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-web-config\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.360042 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.359795 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cda93acf-5e54-430b-8187-b2d6265ff0d0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.360042 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.359813 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.360042 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.359833 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cda93acf-5e54-430b-8187-b2d6265ff0d0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.360042 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.359846 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.360042 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.359936 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cda93acf-5e54-430b-8187-b2d6265ff0d0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.360042 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.359956 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.360042 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.360010 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.360042 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.360027 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda93acf-5e54-430b-8187-b2d6265ff0d0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.360304 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.360044 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cda93acf-5e54-430b-8187-b2d6265ff0d0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.360304 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.360065 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.360304 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.360088 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cda93acf-5e54-430b-8187-b2d6265ff0d0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.460775 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.460737 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cda93acf-5e54-430b-8187-b2d6265ff0d0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.460775 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.460778 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.461024 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.460810 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.461024 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.460834 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda93acf-5e54-430b-8187-b2d6265ff0d0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.461024 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.460887 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cda93acf-5e54-430b-8187-b2d6265ff0d0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.461024 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.460912 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.461024 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.460946 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cda93acf-5e54-430b-8187-b2d6265ff0d0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.461024 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.460985 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-config\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.461024 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.461008 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda93acf-5e54-430b-8187-b2d6265ff0d0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.461334 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.461032 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cda93acf-5e54-430b-8187-b2d6265ff0d0-config-out\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.461334 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.461053 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cda93acf-5e54-430b-8187-b2d6265ff0d0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.461334 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.461078 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2qpx\" (UniqueName: \"kubernetes.io/projected/cda93acf-5e54-430b-8187-b2d6265ff0d0-kube-api-access-z2qpx\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.461334 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.461105 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.461334 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.461132 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.461334 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.461162 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda93acf-5e54-430b-8187-b2d6265ff0d0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.461334 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.461203 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-web-config\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.461334 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.461230 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cda93acf-5e54-430b-8187-b2d6265ff0d0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.461334 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.461257 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.461334 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.461294 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cda93acf-5e54-430b-8187-b2d6265ff0d0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.461334 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.461324 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.461895 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.461661 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cda93acf-5e54-430b-8187-b2d6265ff0d0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.461895 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.461815 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda93acf-5e54-430b-8187-b2d6265ff0d0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.463903 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.463693 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cda93acf-5e54-430b-8187-b2d6265ff0d0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.464027 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.463988 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cda93acf-5e54-430b-8187-b2d6265ff0d0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.465307 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.464336 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.465307 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.464965 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda93acf-5e54-430b-8187-b2d6265ff0d0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.465307 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.465253 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cda93acf-5e54-430b-8187-b2d6265ff0d0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.465307 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.465270 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.465565 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.465436 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.465682 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.465659 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.466147 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.466121 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.466592 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.466569 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.466981 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.466936 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-web-config\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.467090 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.467057 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-config\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.467214 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.467191 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cda93acf-5e54-430b-8187-b2d6265ff0d0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.467264 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.467242 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cda93acf-5e54-430b-8187-b2d6265ff0d0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.467673 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.467649 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cda93acf-5e54-430b-8187-b2d6265ff0d0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.467779 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.467762 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cda93acf-5e54-430b-8187-b2d6265ff0d0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.467924 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.467907 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cda93acf-5e54-430b-8187-b2d6265ff0d0-config-out\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.472264 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.472240 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2qpx\" (UniqueName: \"kubernetes.io/projected/cda93acf-5e54-430b-8187-b2d6265ff0d0-kube-api-access-z2qpx\") pod \"prometheus-k8s-0\" (UID: \"cda93acf-5e54-430b-8187-b2d6265ff0d0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.606760 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.606718 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:13:04.888283 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:04.888256 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 09 23:13:04.890710 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:13:04.890681 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda93acf_5e54_430b_8187_b2d6265ff0d0.slice/crio-edce988198f1019ddff4e94ff0d499b54c2c9cab946d3c80b280f06cf3cdf276 WatchSource:0}: Error finding container edce988198f1019ddff4e94ff0d499b54c2c9cab946d3c80b280f06cf3cdf276: Status 404 returned error can't find the container with id edce988198f1019ddff4e94ff0d499b54c2c9cab946d3c80b280f06cf3cdf276 Sep 09 23:13:05.244373 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:05.244343 2561 generic.go:356] "Generic (PLEG): container finished" podID="cda93acf-5e54-430b-8187-b2d6265ff0d0" containerID="6def0ec6cede042e44afd9d19bd19c6f597020ddebef68324b1f989401d4c899" exitCode=0 Sep 09 23:13:05.244781 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:05.244425 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cda93acf-5e54-430b-8187-b2d6265ff0d0","Type":"ContainerDied","Data":"6def0ec6cede042e44afd9d19bd19c6f597020ddebef68324b1f989401d4c899"} Sep 09 23:13:05.244781 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:05.244496 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cda93acf-5e54-430b-8187-b2d6265ff0d0","Type":"ContainerStarted","Data":"edce988198f1019ddff4e94ff0d499b54c2c9cab946d3c80b280f06cf3cdf276"} Sep 09 23:13:05.810133 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:05.810044 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ba0feeb-146f-4bdf-997d-3ea9d54b25e4" path="/var/lib/kubelet/pods/8ba0feeb-146f-4bdf-997d-3ea9d54b25e4/volumes" Sep 09 23:13:06.253064 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:06.253025 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cda93acf-5e54-430b-8187-b2d6265ff0d0","Type":"ContainerStarted","Data":"e94f1a2c26902e096ca069673f673c769b423a5c5469ed36d47d558300aad93f"} Sep 09 23:13:06.253482 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:06.253098 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cda93acf-5e54-430b-8187-b2d6265ff0d0","Type":"ContainerStarted","Data":"a6d78a99bfb84ef83fd81a6886175e00e9737ea75f2a4eae50dc428fd7585d6e"} Sep 09 23:13:06.253482 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:06.253126 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cda93acf-5e54-430b-8187-b2d6265ff0d0","Type":"ContainerStarted","Data":"b0167218d54feca90528fd1c7adf79c178e45873d7b5243e8f8b36f05897ccf7"} Sep 09 23:13:06.253482 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:06.253150 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cda93acf-5e54-430b-8187-b2d6265ff0d0","Type":"ContainerStarted","Data":"a456aa3067fd50e2cf708c74050299954b0db0240df7b98ba76e4faa9e3ffb1f"} Sep 09 23:13:06.253482 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:06.253173 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cda93acf-5e54-430b-8187-b2d6265ff0d0","Type":"ContainerStarted","Data":"a90dbbda351e5f03aff7f0f940880ec31d107b1a8d8eee76ece5c4ebd5839cb9"} Sep 09 23:13:06.253482 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:06.253196 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cda93acf-5e54-430b-8187-b2d6265ff0d0","Type":"ContainerStarted","Data":"de526f8b41b45de9031480c5d7a28a1749e2a7214c8df56d843ced412e8658e2"} Sep 09 23:13:06.280664 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:06.280609 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.259887694 podStartE2EDuration="2.28059196s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.704266ms" isStatefulPod=false podCreationTimestamp="2026-09-09 23:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:13:06.278637624 +0000 UTC m=+374.964421061" watchObservedRunningTime="2026-09-09 23:13:06.28059196 +0000 UTC m=+374.966375363" Sep 09 23:13:09.606953 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:13:09.606898 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:14:04.607515 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:14:04.607476 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:14:04.624210 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:14:04.624183 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:14:05.517648 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:14:05.517615 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 09 23:16:00.332585 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.332492 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww"] Sep 09 23:16:00.336290 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.336255 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww" Sep 09 23:16:00.338725 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.338693 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 09 23:16:00.339255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.339224 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 09 23:16:00.339553 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.339530 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 09 23:16:00.339806 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.339782 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2wgrm\"" Sep 09 23:16:00.340011 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.339837 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 09 23:16:00.354970 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.354941 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww"] Sep 09 23:16:00.468911 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.468873 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shtq8\" (UniqueName: \"kubernetes.io/projected/b3581d49-a5b5-4b09-96aa-9f58bcc64c9b-kube-api-access-shtq8\") pod \"opendatahub-operator-controller-manager-c5b764bdf-dxhww\" (UID: \"b3581d49-a5b5-4b09-96aa-9f58bcc64c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww" Sep 09 23:16:00.468911 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.468918 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3581d49-a5b5-4b09-96aa-9f58bcc64c9b-tmp\") pod \"opendatahub-operator-controller-manager-c5b764bdf-dxhww\" (UID: \"b3581d49-a5b5-4b09-96aa-9f58bcc64c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww" Sep 09 23:16:00.469141 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.468947 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b3581d49-a5b5-4b09-96aa-9f58bcc64c9b-manifests\") pod \"opendatahub-operator-controller-manager-c5b764bdf-dxhww\" (UID: \"b3581d49-a5b5-4b09-96aa-9f58bcc64c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww" Sep 09 23:16:00.469141 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.468962 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b3581d49-a5b5-4b09-96aa-9f58bcc64c9b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c5b764bdf-dxhww\" (UID: \"b3581d49-a5b5-4b09-96aa-9f58bcc64c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww" Sep 09 23:16:00.469141 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.469036 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3581d49-a5b5-4b09-96aa-9f58bcc64c9b-webhook-cert\") pod \"opendatahub-operator-controller-manager-c5b764bdf-dxhww\" (UID: \"b3581d49-a5b5-4b09-96aa-9f58bcc64c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww" Sep 09 23:16:00.569650 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.569606 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shtq8\" (UniqueName: \"kubernetes.io/projected/b3581d49-a5b5-4b09-96aa-9f58bcc64c9b-kube-api-access-shtq8\") pod \"opendatahub-operator-controller-manager-c5b764bdf-dxhww\" (UID: \"b3581d49-a5b5-4b09-96aa-9f58bcc64c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww" Sep 09 23:16:00.569650 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.569656 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3581d49-a5b5-4b09-96aa-9f58bcc64c9b-tmp\") pod \"opendatahub-operator-controller-manager-c5b764bdf-dxhww\" (UID: \"b3581d49-a5b5-4b09-96aa-9f58bcc64c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww" Sep 09 23:16:00.569967 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.569696 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b3581d49-a5b5-4b09-96aa-9f58bcc64c9b-manifests\") pod \"opendatahub-operator-controller-manager-c5b764bdf-dxhww\" (UID: \"b3581d49-a5b5-4b09-96aa-9f58bcc64c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww" Sep 09 23:16:00.569967 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.569718 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b3581d49-a5b5-4b09-96aa-9f58bcc64c9b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c5b764bdf-dxhww\" (UID: \"b3581d49-a5b5-4b09-96aa-9f58bcc64c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww" Sep 09 23:16:00.569967 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.569762 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3581d49-a5b5-4b09-96aa-9f58bcc64c9b-webhook-cert\") pod \"opendatahub-operator-controller-manager-c5b764bdf-dxhww\" (UID: \"b3581d49-a5b5-4b09-96aa-9f58bcc64c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww" Sep 09 23:16:00.570172 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.570148 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b3581d49-a5b5-4b09-96aa-9f58bcc64c9b-manifests\") pod \"opendatahub-operator-controller-manager-c5b764bdf-dxhww\" (UID: \"b3581d49-a5b5-4b09-96aa-9f58bcc64c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww" Sep 09 23:16:00.572196 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.572174 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3581d49-a5b5-4b09-96aa-9f58bcc64c9b-tmp\") pod \"opendatahub-operator-controller-manager-c5b764bdf-dxhww\" (UID: \"b3581d49-a5b5-4b09-96aa-9f58bcc64c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww" Sep 09 23:16:00.572328 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.572289 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3581d49-a5b5-4b09-96aa-9f58bcc64c9b-webhook-cert\") pod \"opendatahub-operator-controller-manager-c5b764bdf-dxhww\" (UID: \"b3581d49-a5b5-4b09-96aa-9f58bcc64c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww" Sep 09 23:16:00.572499 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.572478 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b3581d49-a5b5-4b09-96aa-9f58bcc64c9b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c5b764bdf-dxhww\" (UID: \"b3581d49-a5b5-4b09-96aa-9f58bcc64c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww" Sep 09 23:16:00.581310 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.581287 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shtq8\" (UniqueName: \"kubernetes.io/projected/b3581d49-a5b5-4b09-96aa-9f58bcc64c9b-kube-api-access-shtq8\") pod \"opendatahub-operator-controller-manager-c5b764bdf-dxhww\" (UID: \"b3581d49-a5b5-4b09-96aa-9f58bcc64c9b\") " pod="opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww" Sep 09 23:16:00.650624 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.650530 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww" Sep 09 23:16:00.941888 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.941170 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww"] Sep 09 23:16:00.944883 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:16:00.944843 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3581d49_a5b5_4b09_96aa_9f58bcc64c9b.slice/crio-cea79cfaaa04cd49b8abe59220a8b34fcac44693ec425950a191cc99719dcce5 WatchSource:0}: Error finding container cea79cfaaa04cd49b8abe59220a8b34fcac44693ec425950a191cc99719dcce5: Status 404 returned error can't find the container with id cea79cfaaa04cd49b8abe59220a8b34fcac44693ec425950a191cc99719dcce5 Sep 09 23:16:00.946720 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.946697 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 23:16:00.956315 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:00.956283 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww" event={"ID":"b3581d49-a5b5-4b09-96aa-9f58bcc64c9b","Type":"ContainerStarted","Data":"cea79cfaaa04cd49b8abe59220a8b34fcac44693ec425950a191cc99719dcce5"} Sep 09 23:16:04.976694 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:04.976665 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww" event={"ID":"b3581d49-a5b5-4b09-96aa-9f58bcc64c9b","Type":"ContainerStarted","Data":"32638327618dc2258f3d790dd88805c6090cfd36de214c99e43aa49cd23d2aeb"} Sep 09 23:16:05.982320 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:05.982282 2561 generic.go:356] "Generic (PLEG): container finished" podID="b3581d49-a5b5-4b09-96aa-9f58bcc64c9b" containerID="32638327618dc2258f3d790dd88805c6090cfd36de214c99e43aa49cd23d2aeb" exitCode=0 Sep 09 23:16:05.982782 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:05.982340 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww" event={"ID":"b3581d49-a5b5-4b09-96aa-9f58bcc64c9b","Type":"ContainerDied","Data":"32638327618dc2258f3d790dd88805c6090cfd36de214c99e43aa49cd23d2aeb"} Sep 09 23:16:06.988803 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:06.988762 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww" event={"ID":"b3581d49-a5b5-4b09-96aa-9f58bcc64c9b","Type":"ContainerStarted","Data":"83d09a0795d58cc527308f71aec078041ff8bc5ecccf07028aa672436b3c84e6"} Sep 09 23:16:06.989236 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:06.988896 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww" Sep 09 23:16:07.011260 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:07.011172 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww" podStartSLOduration=2.609340535 podStartE2EDuration="7.011151133s" totalImagesPullingTime="3.317553853s" totalInitContainerRuntime="1.084256745s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:16:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:16:07.009834872 +0000 UTC m=+555.695618319" watchObservedRunningTime="2026-09-09 23:16:07.011151133 +0000 UTC m=+555.696934537" Sep 09 23:16:17.996588 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:17.996551 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-c5b764bdf-dxhww" Sep 09 23:16:20.409517 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:20.409482 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-66c75bc564-f5ckb"] Sep 09 23:16:20.415961 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:20.415940 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-66c75bc564-f5ckb" Sep 09 23:16:20.418501 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:20.418467 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 09 23:16:20.419466 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:20.419438 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 09 23:16:20.419589 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:20.419509 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 09 23:16:20.419589 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:20.419543 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 09 23:16:20.419589 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:20.419577 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 09 23:16:20.419745 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:20.419684 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-sht6v\"" Sep 09 23:16:20.422016 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:20.421989 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-66c75bc564-f5ckb"] Sep 09 23:16:20.445638 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:20.445598 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a00d2f07-169b-4bbd-9872-9cd3dd60d69f-metrics-cert\") pod \"lws-controller-manager-66c75bc564-f5ckb\" (UID: \"a00d2f07-169b-4bbd-9872-9cd3dd60d69f\") " pod="openshift-lws-operator/lws-controller-manager-66c75bc564-f5ckb" Sep 09 23:16:20.445638 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:20.445642 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a00d2f07-169b-4bbd-9872-9cd3dd60d69f-manager-config\") pod \"lws-controller-manager-66c75bc564-f5ckb\" (UID: \"a00d2f07-169b-4bbd-9872-9cd3dd60d69f\") " pod="openshift-lws-operator/lws-controller-manager-66c75bc564-f5ckb" Sep 09 23:16:20.445638 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:20.445660 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg6lt\" (UniqueName: \"kubernetes.io/projected/a00d2f07-169b-4bbd-9872-9cd3dd60d69f-kube-api-access-cg6lt\") pod \"lws-controller-manager-66c75bc564-f5ckb\" (UID: \"a00d2f07-169b-4bbd-9872-9cd3dd60d69f\") " pod="openshift-lws-operator/lws-controller-manager-66c75bc564-f5ckb" Sep 09 23:16:20.445921 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:20.445799 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a00d2f07-169b-4bbd-9872-9cd3dd60d69f-cert\") pod \"lws-controller-manager-66c75bc564-f5ckb\" (UID: \"a00d2f07-169b-4bbd-9872-9cd3dd60d69f\") " pod="openshift-lws-operator/lws-controller-manager-66c75bc564-f5ckb" Sep 09 23:16:20.546828 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:20.546787 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a00d2f07-169b-4bbd-9872-9cd3dd60d69f-cert\") pod \"lws-controller-manager-66c75bc564-f5ckb\" (UID: \"a00d2f07-169b-4bbd-9872-9cd3dd60d69f\") " pod="openshift-lws-operator/lws-controller-manager-66c75bc564-f5ckb" Sep 09 23:16:20.546828 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:20.546839 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a00d2f07-169b-4bbd-9872-9cd3dd60d69f-metrics-cert\") pod \"lws-controller-manager-66c75bc564-f5ckb\" (UID: \"a00d2f07-169b-4bbd-9872-9cd3dd60d69f\") " pod="openshift-lws-operator/lws-controller-manager-66c75bc564-f5ckb" Sep 09 23:16:20.547130 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:20.546913 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a00d2f07-169b-4bbd-9872-9cd3dd60d69f-manager-config\") pod \"lws-controller-manager-66c75bc564-f5ckb\" (UID: \"a00d2f07-169b-4bbd-9872-9cd3dd60d69f\") " pod="openshift-lws-operator/lws-controller-manager-66c75bc564-f5ckb" Sep 09 23:16:20.547130 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:20.546939 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg6lt\" (UniqueName: \"kubernetes.io/projected/a00d2f07-169b-4bbd-9872-9cd3dd60d69f-kube-api-access-cg6lt\") pod \"lws-controller-manager-66c75bc564-f5ckb\" (UID: \"a00d2f07-169b-4bbd-9872-9cd3dd60d69f\") " pod="openshift-lws-operator/lws-controller-manager-66c75bc564-f5ckb" Sep 09 23:16:20.547629 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:20.547558 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a00d2f07-169b-4bbd-9872-9cd3dd60d69f-manager-config\") pod \"lws-controller-manager-66c75bc564-f5ckb\" (UID: \"a00d2f07-169b-4bbd-9872-9cd3dd60d69f\") " pod="openshift-lws-operator/lws-controller-manager-66c75bc564-f5ckb" Sep 09 23:16:20.550048 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:20.550014 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a00d2f07-169b-4bbd-9872-9cd3dd60d69f-metrics-cert\") pod \"lws-controller-manager-66c75bc564-f5ckb\" (UID: \"a00d2f07-169b-4bbd-9872-9cd3dd60d69f\") " pod="openshift-lws-operator/lws-controller-manager-66c75bc564-f5ckb" Sep 09 23:16:20.550197 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:20.550014 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a00d2f07-169b-4bbd-9872-9cd3dd60d69f-cert\") pod \"lws-controller-manager-66c75bc564-f5ckb\" (UID: \"a00d2f07-169b-4bbd-9872-9cd3dd60d69f\") " pod="openshift-lws-operator/lws-controller-manager-66c75bc564-f5ckb" Sep 09 23:16:20.558847 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:20.558814 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg6lt\" (UniqueName: \"kubernetes.io/projected/a00d2f07-169b-4bbd-9872-9cd3dd60d69f-kube-api-access-cg6lt\") pod \"lws-controller-manager-66c75bc564-f5ckb\" (UID: \"a00d2f07-169b-4bbd-9872-9cd3dd60d69f\") " pod="openshift-lws-operator/lws-controller-manager-66c75bc564-f5ckb" Sep 09 23:16:20.726519 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:20.726486 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-66c75bc564-f5ckb" Sep 09 23:16:20.998925 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:20.998895 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-66c75bc564-f5ckb"] Sep 09 23:16:21.001486 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:16:21.001462 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00d2f07_169b_4bbd_9872_9cd3dd60d69f.slice/crio-15d3f2646049409a020c0cacaba089ef0144ac9312222826e7ad9121443acda4 WatchSource:0}: Error finding container 15d3f2646049409a020c0cacaba089ef0144ac9312222826e7ad9121443acda4: Status 404 returned error can't find the container with id 15d3f2646049409a020c0cacaba089ef0144ac9312222826e7ad9121443acda4 Sep 09 23:16:21.050605 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:21.050565 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-66c75bc564-f5ckb" event={"ID":"a00d2f07-169b-4bbd-9872-9cd3dd60d69f","Type":"ContainerStarted","Data":"15d3f2646049409a020c0cacaba089ef0144ac9312222826e7ad9121443acda4"} Sep 09 23:16:23.782355 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.782308 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74dcdb946c-2czls"] Sep 09 23:16:23.786366 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.786332 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:23.788901 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.788843 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 09 23:16:23.789179 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.789158 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 09 23:16:23.791079 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.789988 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 09 23:16:23.791079 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.790217 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 09 23:16:23.791079 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.790382 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 09 23:16:23.791079 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.790560 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4cpxx\"" Sep 09 23:16:23.791079 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.790736 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 09 23:16:23.791079 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.790931 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 09 23:16:23.796115 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.796089 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74dcdb946c-2czls"] Sep 09 23:16:23.798050 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.797179 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 09 23:16:23.875229 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.875187 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ed8e9b7-ad65-44eb-a256-f23db3a47011-trusted-ca-bundle\") pod \"console-74dcdb946c-2czls\" (UID: \"6ed8e9b7-ad65-44eb-a256-f23db3a47011\") " pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:23.875229 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.875236 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed8e9b7-ad65-44eb-a256-f23db3a47011-service-ca\") pod \"console-74dcdb946c-2czls\" (UID: \"6ed8e9b7-ad65-44eb-a256-f23db3a47011\") " pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:23.875478 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.875265 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ed8e9b7-ad65-44eb-a256-f23db3a47011-console-serving-cert\") pod \"console-74dcdb946c-2czls\" (UID: \"6ed8e9b7-ad65-44eb-a256-f23db3a47011\") " pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:23.875478 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.875337 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqgsq\" (UniqueName: \"kubernetes.io/projected/6ed8e9b7-ad65-44eb-a256-f23db3a47011-kube-api-access-kqgsq\") pod \"console-74dcdb946c-2czls\" (UID: \"6ed8e9b7-ad65-44eb-a256-f23db3a47011\") " pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:23.875478 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.875363 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6ed8e9b7-ad65-44eb-a256-f23db3a47011-console-oauth-config\") pod \"console-74dcdb946c-2czls\" (UID: \"6ed8e9b7-ad65-44eb-a256-f23db3a47011\") " pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:23.875478 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.875379 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6ed8e9b7-ad65-44eb-a256-f23db3a47011-oauth-serving-cert\") pod \"console-74dcdb946c-2czls\" (UID: \"6ed8e9b7-ad65-44eb-a256-f23db3a47011\") " pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:23.875478 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.875398 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6ed8e9b7-ad65-44eb-a256-f23db3a47011-console-config\") pod \"console-74dcdb946c-2czls\" (UID: \"6ed8e9b7-ad65-44eb-a256-f23db3a47011\") " pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:23.976740 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.976698 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqgsq\" (UniqueName: \"kubernetes.io/projected/6ed8e9b7-ad65-44eb-a256-f23db3a47011-kube-api-access-kqgsq\") pod \"console-74dcdb946c-2czls\" (UID: \"6ed8e9b7-ad65-44eb-a256-f23db3a47011\") " pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:23.976740 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.976741 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6ed8e9b7-ad65-44eb-a256-f23db3a47011-console-oauth-config\") pod \"console-74dcdb946c-2czls\" (UID: \"6ed8e9b7-ad65-44eb-a256-f23db3a47011\") " pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:23.977066 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.976767 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6ed8e9b7-ad65-44eb-a256-f23db3a47011-oauth-serving-cert\") pod \"console-74dcdb946c-2czls\" (UID: \"6ed8e9b7-ad65-44eb-a256-f23db3a47011\") " pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:23.977066 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.976799 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6ed8e9b7-ad65-44eb-a256-f23db3a47011-console-config\") pod \"console-74dcdb946c-2czls\" (UID: \"6ed8e9b7-ad65-44eb-a256-f23db3a47011\") " pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:23.977066 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.976826 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ed8e9b7-ad65-44eb-a256-f23db3a47011-trusted-ca-bundle\") pod \"console-74dcdb946c-2czls\" (UID: \"6ed8e9b7-ad65-44eb-a256-f23db3a47011\") " pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:23.977066 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.976900 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed8e9b7-ad65-44eb-a256-f23db3a47011-service-ca\") pod \"console-74dcdb946c-2czls\" (UID: \"6ed8e9b7-ad65-44eb-a256-f23db3a47011\") " pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:23.977066 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.976929 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ed8e9b7-ad65-44eb-a256-f23db3a47011-console-serving-cert\") pod \"console-74dcdb946c-2czls\" (UID: \"6ed8e9b7-ad65-44eb-a256-f23db3a47011\") " pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:23.977693 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.977668 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6ed8e9b7-ad65-44eb-a256-f23db3a47011-console-config\") pod \"console-74dcdb946c-2czls\" (UID: \"6ed8e9b7-ad65-44eb-a256-f23db3a47011\") " pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:23.977814 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.977668 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6ed8e9b7-ad65-44eb-a256-f23db3a47011-oauth-serving-cert\") pod \"console-74dcdb946c-2czls\" (UID: \"6ed8e9b7-ad65-44eb-a256-f23db3a47011\") " pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:23.977814 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.977668 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed8e9b7-ad65-44eb-a256-f23db3a47011-service-ca\") pod \"console-74dcdb946c-2czls\" (UID: \"6ed8e9b7-ad65-44eb-a256-f23db3a47011\") " pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:23.978674 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.978647 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ed8e9b7-ad65-44eb-a256-f23db3a47011-trusted-ca-bundle\") pod \"console-74dcdb946c-2czls\" (UID: \"6ed8e9b7-ad65-44eb-a256-f23db3a47011\") " pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:23.979348 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.979327 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6ed8e9b7-ad65-44eb-a256-f23db3a47011-console-oauth-config\") pod \"console-74dcdb946c-2czls\" (UID: \"6ed8e9b7-ad65-44eb-a256-f23db3a47011\") " pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:23.979725 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.979705 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ed8e9b7-ad65-44eb-a256-f23db3a47011-console-serving-cert\") pod \"console-74dcdb946c-2czls\" (UID: \"6ed8e9b7-ad65-44eb-a256-f23db3a47011\") " pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:23.984030 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:23.984005 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqgsq\" (UniqueName: \"kubernetes.io/projected/6ed8e9b7-ad65-44eb-a256-f23db3a47011-kube-api-access-kqgsq\") pod \"console-74dcdb946c-2czls\" (UID: \"6ed8e9b7-ad65-44eb-a256-f23db3a47011\") " pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:24.066204 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:24.066105 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-66c75bc564-f5ckb" event={"ID":"a00d2f07-169b-4bbd-9872-9cd3dd60d69f","Type":"ContainerStarted","Data":"cbb5f39e8d6233bcf43d0f90a2cefa1870538cfaf4b74369d3c2048f89caa226"} Sep 09 23:16:24.066368 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:24.066204 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-66c75bc564-f5ckb" Sep 09 23:16:24.081300 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:24.081241 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-66c75bc564-f5ckb" podStartSLOduration=1.850381897 podStartE2EDuration="4.081222477s" totalImagesPullingTime="2.23084058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:16:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:16:24.080260741 +0000 UTC m=+572.766044161" watchObservedRunningTime="2026-09-09 23:16:24.081222477 +0000 UTC m=+572.767005879" Sep 09 23:16:24.102378 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:24.102341 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:24.378585 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:24.378479 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74dcdb946c-2czls"] Sep 09 23:16:24.381872 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:16:24.381825 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed8e9b7_ad65_44eb_a256_f23db3a47011.slice/crio-c73fec97bd2d109f3e77fc5afe5646d3450b14956f3535c9e2d6ba2ef2fa83e6 WatchSource:0}: Error finding container c73fec97bd2d109f3e77fc5afe5646d3450b14956f3535c9e2d6ba2ef2fa83e6: Status 404 returned error can't find the container with id c73fec97bd2d109f3e77fc5afe5646d3450b14956f3535c9e2d6ba2ef2fa83e6 Sep 09 23:16:25.072294 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:25.072250 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74dcdb946c-2czls" event={"ID":"6ed8e9b7-ad65-44eb-a256-f23db3a47011","Type":"ContainerStarted","Data":"bce64b1f222e061bccae9973c72b513915a5b6e58164c680580d9b0f3b66c582"} Sep 09 23:16:25.072676 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:25.072336 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74dcdb946c-2czls" event={"ID":"6ed8e9b7-ad65-44eb-a256-f23db3a47011","Type":"ContainerStarted","Data":"c73fec97bd2d109f3e77fc5afe5646d3450b14956f3535c9e2d6ba2ef2fa83e6"} Sep 09 23:16:25.091168 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:25.091104 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74dcdb946c-2czls" podStartSLOduration=2.091082017 podStartE2EDuration="2.091082017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:16:25.08690006 +0000 UTC m=+573.772683471" watchObservedRunningTime="2026-09-09 23:16:25.091082017 +0000 UTC m=+573.776865421" Sep 09 23:16:34.103256 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:34.103203 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:34.103745 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:34.103280 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:34.108478 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:34.108436 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:34.113219 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:34.113191 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74dcdb946c-2czls" Sep 09 23:16:35.074685 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:35.074651 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-66c75bc564-f5ckb" Sep 09 23:16:37.886418 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:37.886378 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf"] Sep 09 23:16:37.891069 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:37.891049 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" Sep 09 23:16:37.893656 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:37.893620 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 09 23:16:37.893848 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:37.893660 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 09 23:16:37.893848 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:37.893670 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-j8fbt\"" Sep 09 23:16:37.893848 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:37.893728 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 09 23:16:37.894020 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:37.893960 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 09 23:16:37.900713 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:37.900680 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf"] Sep 09 23:16:38.005526 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:38.005486 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-jnwpf\" (UID: \"5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" Sep 09 23:16:38.005526 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:38.005528 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-jnwpf\" (UID: \"5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" Sep 09 23:16:38.005752 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:38.005562 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-jnwpf\" (UID: \"5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" Sep 09 23:16:38.005752 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:38.005626 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-jnwpf\" (UID: \"5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" Sep 09 23:16:38.005752 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:38.005669 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-jnwpf\" (UID: \"5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" Sep 09 23:16:38.005752 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:38.005695 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-jnwpf\" (UID: \"5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" Sep 09 23:16:38.005752 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:38.005710 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kd9m\" (UniqueName: \"kubernetes.io/projected/5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae-kube-api-access-4kd9m\") pod \"istiod-openshift-gateway-5677ccb49b-jnwpf\" (UID: \"5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" Sep 09 23:16:38.107019 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:38.106977 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-jnwpf\" (UID: \"5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" Sep 09 23:16:38.107019 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:38.107018 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kd9m\" (UniqueName: \"kubernetes.io/projected/5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae-kube-api-access-4kd9m\") pod \"istiod-openshift-gateway-5677ccb49b-jnwpf\" (UID: \"5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" Sep 09 23:16:38.107287 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:38.107067 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-jnwpf\" (UID: \"5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" Sep 09 23:16:38.107287 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:38.107089 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-jnwpf\" (UID: \"5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" Sep 09 23:16:38.107287 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:38.107121 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-jnwpf\" (UID: \"5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" Sep 09 23:16:38.107287 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:38.107164 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-jnwpf\" (UID: \"5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" Sep 09 23:16:38.107287 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:38.107211 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-jnwpf\" (UID: \"5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" Sep 09 23:16:38.108085 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:38.108058 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-jnwpf\" (UID: \"5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" Sep 09 23:16:38.109661 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:38.109639 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-jnwpf\" (UID: \"5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" Sep 09 23:16:38.109760 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:38.109658 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-jnwpf\" (UID: \"5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" Sep 09 23:16:38.110132 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:38.110109 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-jnwpf\" (UID: \"5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" Sep 09 23:16:38.110233 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:38.110152 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-jnwpf\" (UID: \"5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" Sep 09 23:16:38.118693 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:38.118670 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-jnwpf\" (UID: \"5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" Sep 09 23:16:38.118893 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:38.118873 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kd9m\" (UniqueName: \"kubernetes.io/projected/5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae-kube-api-access-4kd9m\") pod \"istiod-openshift-gateway-5677ccb49b-jnwpf\" (UID: \"5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" Sep 09 23:16:38.202038 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:38.202011 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" Sep 09 23:16:38.506911 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:38.506796 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf"] Sep 09 23:16:38.510611 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:16:38.510570 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a431b8e_ee49_45e1_aa0b_5f67d97ff7ae.slice/crio-89462d690d585867d383b7540ad9119f586d1ddacc5726f3e289869e816764cb WatchSource:0}: Error finding container 89462d690d585867d383b7540ad9119f586d1ddacc5726f3e289869e816764cb: Status 404 returned error can't find the container with id 89462d690d585867d383b7540ad9119f586d1ddacc5726f3e289869e816764cb Sep 09 23:16:39.136215 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:39.136172 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" event={"ID":"5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae","Type":"ContainerStarted","Data":"89462d690d585867d383b7540ad9119f586d1ddacc5726f3e289869e816764cb"} Sep 09 23:16:40.936744 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:40.936703 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Sep 09 23:16:40.937094 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:40.936772 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Sep 09 23:16:41.146808 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:41.146769 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" event={"ID":"5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae","Type":"ContainerStarted","Data":"ad88f58f8c1e333950cf4ce6a6e4e94d7c46f2edaaf7f5500ea07a786480e24d"} Sep 09 23:16:41.147033 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:41.146993 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" Sep 09 23:16:41.148834 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:41.148804 2561 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-jnwpf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 09 23:16:41.148980 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:41.148864 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" podUID="5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 09 23:16:41.214453 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:41.214295 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" podStartSLOduration=1.790926027 podStartE2EDuration="4.214275362s" totalImagesPullingTime="2.423349335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:16:41.212425436 +0000 UTC m=+589.898208861" watchObservedRunningTime="2026-09-09 23:16:41.214275362 +0000 UTC m=+589.900058765" Sep 09 23:16:42.151793 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:42.151749 2561 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-jnwpf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 09 23:16:42.152263 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:42.151828 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" podUID="5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 09 23:16:45.152258 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:45.152223 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jnwpf" Sep 09 23:16:51.730683 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:51.730649 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2fs7f_0a127828-4a97-40cf-b8ae-f5ecd20f97c6/console-operator/1.log" Sep 09 23:16:51.731154 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:16:51.730649 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2fs7f_0a127828-4a97-40cf-b8ae-f5ecd20f97c6/console-operator/1.log" Sep 09 23:18:10.207335 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:10.207303 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ltvgp"] Sep 09 23:18:10.210691 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:10.210673 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ltvgp" Sep 09 23:18:10.213143 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:10.213123 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 09 23:18:10.213143 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:10.213125 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 09 23:18:10.213345 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:10.213228 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5mct6\"" Sep 09 23:18:10.226489 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:10.226462 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ltvgp"] Sep 09 23:18:10.330829 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:10.330786 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm4qb\" (UniqueName: \"kubernetes.io/projected/d12c537f-7068-4818-92b0-d7c121241c68-kube-api-access-nm4qb\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-ltvgp\" (UID: \"d12c537f-7068-4818-92b0-d7c121241c68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ltvgp" Sep 09 23:18:10.331048 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:10.330846 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d12c537f-7068-4818-92b0-d7c121241c68-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-ltvgp\" (UID: \"d12c537f-7068-4818-92b0-d7c121241c68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ltvgp" Sep 09 23:18:10.431949 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:10.431904 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d12c537f-7068-4818-92b0-d7c121241c68-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-ltvgp\" (UID: \"d12c537f-7068-4818-92b0-d7c121241c68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ltvgp" Sep 09 23:18:10.432161 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:10.432028 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm4qb\" (UniqueName: \"kubernetes.io/projected/d12c537f-7068-4818-92b0-d7c121241c68-kube-api-access-nm4qb\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-ltvgp\" (UID: \"d12c537f-7068-4818-92b0-d7c121241c68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ltvgp" Sep 09 23:18:10.432303 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:10.432284 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d12c537f-7068-4818-92b0-d7c121241c68-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-ltvgp\" (UID: \"d12c537f-7068-4818-92b0-d7c121241c68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ltvgp" Sep 09 23:18:10.440956 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:10.440928 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm4qb\" (UniqueName: \"kubernetes.io/projected/d12c537f-7068-4818-92b0-d7c121241c68-kube-api-access-nm4qb\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-ltvgp\" (UID: \"d12c537f-7068-4818-92b0-d7c121241c68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ltvgp" Sep 09 23:18:10.521302 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:10.521224 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ltvgp" Sep 09 23:18:10.799364 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:10.799169 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ltvgp"] Sep 09 23:18:10.802733 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:18:10.802701 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12c537f_7068_4818_92b0_d7c121241c68.slice/crio-e8b8245ce050eb15a5b5d453bd22cc1f7b548edfc7a1b03ebc9e9104b30e9fa9 WatchSource:0}: Error finding container e8b8245ce050eb15a5b5d453bd22cc1f7b548edfc7a1b03ebc9e9104b30e9fa9: Status 404 returned error can't find the container with id e8b8245ce050eb15a5b5d453bd22cc1f7b548edfc7a1b03ebc9e9104b30e9fa9 Sep 09 23:18:11.212042 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:11.212007 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ltvgp"] Sep 09 23:18:11.216718 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:11.216691 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ltvgp"] Sep 09 23:18:11.240190 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:11.240153 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm4x9"] Sep 09 23:18:11.244773 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:11.244748 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm4x9" Sep 09 23:18:11.258424 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:11.258393 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm4x9"] Sep 09 23:18:11.341519 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:11.341478 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfm2j\" (UniqueName: \"kubernetes.io/projected/fcbf177b-811a-42b1-a02e-ebb911437864-kube-api-access-dfm2j\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-jm4x9\" (UID: \"fcbf177b-811a-42b1-a02e-ebb911437864\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm4x9" Sep 09 23:18:11.341721 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:11.341528 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fcbf177b-811a-42b1-a02e-ebb911437864-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-jm4x9\" (UID: \"fcbf177b-811a-42b1-a02e-ebb911437864\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm4x9" Sep 09 23:18:11.442726 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:11.442670 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fcbf177b-811a-42b1-a02e-ebb911437864-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-jm4x9\" (UID: \"fcbf177b-811a-42b1-a02e-ebb911437864\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm4x9" Sep 09 23:18:11.442967 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:11.442827 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfm2j\" (UniqueName: \"kubernetes.io/projected/fcbf177b-811a-42b1-a02e-ebb911437864-kube-api-access-dfm2j\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-jm4x9\" (UID: \"fcbf177b-811a-42b1-a02e-ebb911437864\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm4x9" Sep 09 23:18:11.443142 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:11.443118 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fcbf177b-811a-42b1-a02e-ebb911437864-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-jm4x9\" (UID: \"fcbf177b-811a-42b1-a02e-ebb911437864\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm4x9" Sep 09 23:18:11.457665 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:11.457635 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfm2j\" (UniqueName: \"kubernetes.io/projected/fcbf177b-811a-42b1-a02e-ebb911437864-kube-api-access-dfm2j\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-jm4x9\" (UID: \"fcbf177b-811a-42b1-a02e-ebb911437864\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm4x9" Sep 09 23:18:11.559762 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:11.559670 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm4x9" Sep 09 23:18:11.858573 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:11.858543 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm4x9"] Sep 09 23:18:11.862193 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:18:11.862157 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcbf177b_811a_42b1_a02e_ebb911437864.slice/crio-1c59a73cf197128c95e36eb38ec8bf0e60779d1a0d65b73ee9d03f83a7535af2 WatchSource:0}: Error finding container 1c59a73cf197128c95e36eb38ec8bf0e60779d1a0d65b73ee9d03f83a7535af2: Status 404 returned error can't find the container with id 1c59a73cf197128c95e36eb38ec8bf0e60779d1a0d65b73ee9d03f83a7535af2 Sep 09 23:18:12.597021 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:12.596976 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm4x9" event={"ID":"fcbf177b-811a-42b1-a02e-ebb911437864","Type":"ContainerStarted","Data":"1c59a73cf197128c95e36eb38ec8bf0e60779d1a0d65b73ee9d03f83a7535af2"} Sep 09 23:18:15.614053 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:15.613963 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm4x9" event={"ID":"fcbf177b-811a-42b1-a02e-ebb911437864","Type":"ContainerStarted","Data":"8b20b86df31b1e2ce73d91b9005be375b2766646057cf09da5b1f3971022c382"} Sep 09 23:18:15.614508 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:15.614063 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm4x9" Sep 09 23:18:15.615929 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:15.615901 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ltvgp" podUID="d12c537f-7068-4818-92b0-d7c121241c68" containerName="manager" containerID="cri-o://183d94950509083463e11c81a691ef2a07e4d8ed02f46b1cc201977c27e2ed6a" gracePeriod=2 Sep 09 23:18:15.635205 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:15.635155 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm4x9" podStartSLOduration=1.19081931 podStartE2EDuration="4.635137992s" totalImagesPullingTime="3.444318682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:18:15.632671946 +0000 UTC m=+684.318455367" watchObservedRunningTime="2026-09-09 23:18:15.635137992 +0000 UTC m=+684.320921395" Sep 09 23:18:15.659163 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:18:15.659124 2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12c537f_7068_4818_92b0_d7c121241c68.slice/crio-conmon-183d94950509083463e11c81a691ef2a07e4d8ed02f46b1cc201977c27e2ed6a.scope\": RecentStats: unable to find data in memory cache]" Sep 09 23:18:15.857564 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:15.857521 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ltvgp" Sep 09 23:18:15.986255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:15.986193 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nm4qb\" (UniqueName: \"kubernetes.io/projected/d12c537f-7068-4818-92b0-d7c121241c68-kube-api-access-nm4qb\") pod \"d12c537f-7068-4818-92b0-d7c121241c68\" (UID: \"d12c537f-7068-4818-92b0-d7c121241c68\") " Sep 09 23:18:15.986437 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:15.986350 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d12c537f-7068-4818-92b0-d7c121241c68-extensions-socket-volume\") pod \"d12c537f-7068-4818-92b0-d7c121241c68\" (UID: \"d12c537f-7068-4818-92b0-d7c121241c68\") " Sep 09 23:18:15.986634 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:15.986609 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12c537f-7068-4818-92b0-d7c121241c68-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d12c537f-7068-4818-92b0-d7c121241c68" (UID: "d12c537f-7068-4818-92b0-d7c121241c68"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:18:15.988765 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:15.988732 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12c537f-7068-4818-92b0-d7c121241c68-kube-api-access-nm4qb" (OuterVolumeSpecName: "kube-api-access-nm4qb") pod "d12c537f-7068-4818-92b0-d7c121241c68" (UID: "d12c537f-7068-4818-92b0-d7c121241c68"). InnerVolumeSpecName "kube-api-access-nm4qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:18:16.087022 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:16.086995 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d12c537f-7068-4818-92b0-d7c121241c68-extensions-socket-volume\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:18:16.087022 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:16.087021 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm4qb\" (UniqueName: \"kubernetes.io/projected/d12c537f-7068-4818-92b0-d7c121241c68-kube-api-access-nm4qb\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:18:16.622048 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:16.621949 2561 generic.go:356] "Generic (PLEG): container finished" podID="d12c537f-7068-4818-92b0-d7c121241c68" containerID="183d94950509083463e11c81a691ef2a07e4d8ed02f46b1cc201977c27e2ed6a" exitCode=2 Sep 09 23:18:16.622048 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:16.622001 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ltvgp" Sep 09 23:18:16.622541 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:16.622049 2561 scope.go:117] "RemoveContainer" containerID="183d94950509083463e11c81a691ef2a07e4d8ed02f46b1cc201977c27e2ed6a" Sep 09 23:18:16.624113 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:18:16.624085 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-ltvgp\" is forbidden: User \"system:node:ip-10-0-143-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-192.ec2.internal' and this object" podUID="d12c537f-7068-4818-92b0-d7c121241c68" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ltvgp" Sep 09 23:18:16.637412 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:18:16.637377 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-ltvgp\" is forbidden: User \"system:node:ip-10-0-143-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-192.ec2.internal' and this object" podUID="d12c537f-7068-4818-92b0-d7c121241c68" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ltvgp" Sep 09 23:18:16.638122 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:16.638100 2561 scope.go:117] "RemoveContainer" containerID="183d94950509083463e11c81a691ef2a07e4d8ed02f46b1cc201977c27e2ed6a" Sep 09 23:18:16.638433 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:18:16.638413 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"183d94950509083463e11c81a691ef2a07e4d8ed02f46b1cc201977c27e2ed6a\": container with ID starting with 183d94950509083463e11c81a691ef2a07e4d8ed02f46b1cc201977c27e2ed6a not found: ID does not exist" containerID="183d94950509083463e11c81a691ef2a07e4d8ed02f46b1cc201977c27e2ed6a" Sep 09 23:18:16.638494 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:16.638441 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"183d94950509083463e11c81a691ef2a07e4d8ed02f46b1cc201977c27e2ed6a"} err="failed to get container status \"183d94950509083463e11c81a691ef2a07e4d8ed02f46b1cc201977c27e2ed6a\": rpc error: code = NotFound desc = could not find container \"183d94950509083463e11c81a691ef2a07e4d8ed02f46b1cc201977c27e2ed6a\": container with ID starting with 183d94950509083463e11c81a691ef2a07e4d8ed02f46b1cc201977c27e2ed6a not found: ID does not exist" Sep 09 23:18:17.809600 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:17.809566 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d12c537f-7068-4818-92b0-d7c121241c68" path="/var/lib/kubelet/pods/d12c537f-7068-4818-92b0-d7c121241c68/volumes" Sep 09 23:18:26.624716 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:26.624677 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm4x9" Sep 09 23:18:39.314833 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:39.314785 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm4x9"] Sep 09 23:18:39.315354 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:39.315082 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm4x9" podUID="fcbf177b-811a-42b1-a02e-ebb911437864" containerName="manager" containerID="cri-o://8b20b86df31b1e2ce73d91b9005be375b2766646057cf09da5b1f3971022c382" gracePeriod=10 Sep 09 23:18:39.568518 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:39.568456 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm4x9" Sep 09 23:18:39.605921 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:39.605888 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fcbf177b-811a-42b1-a02e-ebb911437864-extensions-socket-volume\") pod \"fcbf177b-811a-42b1-a02e-ebb911437864\" (UID: \"fcbf177b-811a-42b1-a02e-ebb911437864\") " Sep 09 23:18:39.606120 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:39.605953 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfm2j\" (UniqueName: \"kubernetes.io/projected/fcbf177b-811a-42b1-a02e-ebb911437864-kube-api-access-dfm2j\") pod \"fcbf177b-811a-42b1-a02e-ebb911437864\" (UID: \"fcbf177b-811a-42b1-a02e-ebb911437864\") " Sep 09 23:18:39.606419 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:39.606388 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcbf177b-811a-42b1-a02e-ebb911437864-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fcbf177b-811a-42b1-a02e-ebb911437864" (UID: "fcbf177b-811a-42b1-a02e-ebb911437864"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:18:39.608467 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:39.608431 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcbf177b-811a-42b1-a02e-ebb911437864-kube-api-access-dfm2j" (OuterVolumeSpecName: "kube-api-access-dfm2j") pod "fcbf177b-811a-42b1-a02e-ebb911437864" (UID: "fcbf177b-811a-42b1-a02e-ebb911437864"). InnerVolumeSpecName "kube-api-access-dfm2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:18:39.707364 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:39.707325 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfm2j\" (UniqueName: \"kubernetes.io/projected/fcbf177b-811a-42b1-a02e-ebb911437864-kube-api-access-dfm2j\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:18:39.707364 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:39.707355 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fcbf177b-811a-42b1-a02e-ebb911437864-extensions-socket-volume\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:18:39.731337 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:39.731299 2561 generic.go:356] "Generic (PLEG): container finished" podID="fcbf177b-811a-42b1-a02e-ebb911437864" containerID="8b20b86df31b1e2ce73d91b9005be375b2766646057cf09da5b1f3971022c382" exitCode=0 Sep 09 23:18:39.731519 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:39.731345 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm4x9" Sep 09 23:18:39.731519 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:39.731379 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm4x9" event={"ID":"fcbf177b-811a-42b1-a02e-ebb911437864","Type":"ContainerDied","Data":"8b20b86df31b1e2ce73d91b9005be375b2766646057cf09da5b1f3971022c382"} Sep 09 23:18:39.731519 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:39.731434 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm4x9" event={"ID":"fcbf177b-811a-42b1-a02e-ebb911437864","Type":"ContainerDied","Data":"1c59a73cf197128c95e36eb38ec8bf0e60779d1a0d65b73ee9d03f83a7535af2"} Sep 09 23:18:39.731519 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:39.731457 2561 scope.go:117] "RemoveContainer" containerID="8b20b86df31b1e2ce73d91b9005be375b2766646057cf09da5b1f3971022c382" Sep 09 23:18:39.740722 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:39.740701 2561 scope.go:117] "RemoveContainer" containerID="8b20b86df31b1e2ce73d91b9005be375b2766646057cf09da5b1f3971022c382" Sep 09 23:18:39.741036 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:18:39.741018 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b20b86df31b1e2ce73d91b9005be375b2766646057cf09da5b1f3971022c382\": container with ID starting with 8b20b86df31b1e2ce73d91b9005be375b2766646057cf09da5b1f3971022c382 not found: ID does not exist" containerID="8b20b86df31b1e2ce73d91b9005be375b2766646057cf09da5b1f3971022c382" Sep 09 23:18:39.741093 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:39.741047 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b20b86df31b1e2ce73d91b9005be375b2766646057cf09da5b1f3971022c382"} err="failed to get container status \"8b20b86df31b1e2ce73d91b9005be375b2766646057cf09da5b1f3971022c382\": rpc error: code = NotFound desc = could not find container \"8b20b86df31b1e2ce73d91b9005be375b2766646057cf09da5b1f3971022c382\": container with ID starting with 8b20b86df31b1e2ce73d91b9005be375b2766646057cf09da5b1f3971022c382 not found: ID does not exist" Sep 09 23:18:39.754870 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:39.754834 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm4x9"] Sep 09 23:18:39.759298 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:39.759269 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm4x9"] Sep 09 23:18:39.809402 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:39.809372 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcbf177b-811a-42b1-a02e-ebb911437864" path="/var/lib/kubelet/pods/fcbf177b-811a-42b1-a02e-ebb911437864/volumes" Sep 09 23:18:58.395940 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:58.395904 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-fjwlz"] Sep 09 23:18:58.396345 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:58.396254 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12c537f-7068-4818-92b0-d7c121241c68" containerName="manager" Sep 09 23:18:58.396345 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:58.396264 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12c537f-7068-4818-92b0-d7c121241c68" containerName="manager" Sep 09 23:18:58.396345 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:58.396274 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcbf177b-811a-42b1-a02e-ebb911437864" containerName="manager" Sep 09 23:18:58.396345 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:58.396279 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcbf177b-811a-42b1-a02e-ebb911437864" containerName="manager" Sep 09 23:18:58.396345 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:58.396328 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcbf177b-811a-42b1-a02e-ebb911437864" containerName="manager" Sep 09 23:18:58.396345 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:58.396338 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d12c537f-7068-4818-92b0-d7c121241c68" containerName="manager" Sep 09 23:18:58.399446 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:58.399428 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-fjwlz" Sep 09 23:18:58.401699 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:58.401671 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 09 23:18:58.401893 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:58.401736 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 09 23:18:58.401893 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:58.401742 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 09 23:18:58.402390 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:58.402372 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-v9t29\"" Sep 09 23:18:58.407508 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:58.407486 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-fjwlz"] Sep 09 23:18:58.482947 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:58.482904 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/04c9b7e4-6886-4d0b-abda-d0995ac30842-config-file\") pod \"limitador-limitador-8588749977-fjwlz\" (UID: \"04c9b7e4-6886-4d0b-abda-d0995ac30842\") " pod="kuadrant-system/limitador-limitador-8588749977-fjwlz" Sep 09 23:18:58.483118 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:58.483083 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6bsz\" (UniqueName: \"kubernetes.io/projected/04c9b7e4-6886-4d0b-abda-d0995ac30842-kube-api-access-m6bsz\") pod \"limitador-limitador-8588749977-fjwlz\" (UID: \"04c9b7e4-6886-4d0b-abda-d0995ac30842\") " pod="kuadrant-system/limitador-limitador-8588749977-fjwlz" Sep 09 23:18:58.498667 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:58.498632 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-fjwlz"] Sep 09 23:18:58.583605 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:58.583565 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/04c9b7e4-6886-4d0b-abda-d0995ac30842-config-file\") pod \"limitador-limitador-8588749977-fjwlz\" (UID: \"04c9b7e4-6886-4d0b-abda-d0995ac30842\") " pod="kuadrant-system/limitador-limitador-8588749977-fjwlz" Sep 09 23:18:58.583772 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:58.583669 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6bsz\" (UniqueName: \"kubernetes.io/projected/04c9b7e4-6886-4d0b-abda-d0995ac30842-kube-api-access-m6bsz\") pod \"limitador-limitador-8588749977-fjwlz\" (UID: \"04c9b7e4-6886-4d0b-abda-d0995ac30842\") " pod="kuadrant-system/limitador-limitador-8588749977-fjwlz" Sep 09 23:18:58.584292 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:58.584260 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/04c9b7e4-6886-4d0b-abda-d0995ac30842-config-file\") pod \"limitador-limitador-8588749977-fjwlz\" (UID: \"04c9b7e4-6886-4d0b-abda-d0995ac30842\") " pod="kuadrant-system/limitador-limitador-8588749977-fjwlz" Sep 09 23:18:58.591178 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:58.591146 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6bsz\" (UniqueName: \"kubernetes.io/projected/04c9b7e4-6886-4d0b-abda-d0995ac30842-kube-api-access-m6bsz\") pod \"limitador-limitador-8588749977-fjwlz\" (UID: \"04c9b7e4-6886-4d0b-abda-d0995ac30842\") " pod="kuadrant-system/limitador-limitador-8588749977-fjwlz" Sep 09 23:18:58.710799 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:58.710766 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-fjwlz" Sep 09 23:18:58.982959 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:58.982565 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-fjwlz"] Sep 09 23:18:58.987007 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:18:58.986960 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c9b7e4_6886_4d0b_abda_d0995ac30842.slice/crio-f818ef3c438b1637008ea84fe3c610f27bf59f3de2c054cfe8bb68521a5a472e WatchSource:0}: Error finding container f818ef3c438b1637008ea84fe3c610f27bf59f3de2c054cfe8bb68521a5a472e: Status 404 returned error can't find the container with id f818ef3c438b1637008ea84fe3c610f27bf59f3de2c054cfe8bb68521a5a472e Sep 09 23:18:59.833251 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:18:59.833215 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-fjwlz" event={"ID":"04c9b7e4-6886-4d0b-abda-d0995ac30842","Type":"ContainerStarted","Data":"f818ef3c438b1637008ea84fe3c610f27bf59f3de2c054cfe8bb68521a5a472e"} Sep 09 23:19:00.839508 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:00.839469 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-fjwlz" event={"ID":"04c9b7e4-6886-4d0b-abda-d0995ac30842","Type":"ContainerStarted","Data":"27342608d851bff1e5fec20c57062e4d534185e969afba91100b7a2596872719"} Sep 09 23:19:00.839974 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:00.839608 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-fjwlz" Sep 09 23:19:00.854365 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:00.854313 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-fjwlz" podStartSLOduration=1.537561792 podStartE2EDuration="2.854295432s" totalImagesPullingTime="1.31673364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:18:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:19:00.853303597 +0000 UTC m=+729.539087023" watchObservedRunningTime="2026-09-09 23:19:00.854295432 +0000 UTC m=+729.540078826" Sep 09 23:19:11.845131 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:11.845092 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-fjwlz" Sep 09 23:19:12.142303 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:12.142200 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-fjwlz"] Sep 09 23:19:12.142502 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:12.142448 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-fjwlz" podUID="04c9b7e4-6886-4d0b-abda-d0995ac30842" containerName="limitador" containerID="cri-o://27342608d851bff1e5fec20c57062e4d534185e969afba91100b7a2596872719" gracePeriod=30 Sep 09 23:19:12.689542 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:12.689514 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-fjwlz" Sep 09 23:19:12.813766 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:12.813676 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6bsz\" (UniqueName: \"kubernetes.io/projected/04c9b7e4-6886-4d0b-abda-d0995ac30842-kube-api-access-m6bsz\") pod \"04c9b7e4-6886-4d0b-abda-d0995ac30842\" (UID: \"04c9b7e4-6886-4d0b-abda-d0995ac30842\") " Sep 09 23:19:12.813969 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:12.813785 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/04c9b7e4-6886-4d0b-abda-d0995ac30842-config-file\") pod \"04c9b7e4-6886-4d0b-abda-d0995ac30842\" (UID: \"04c9b7e4-6886-4d0b-abda-d0995ac30842\") " Sep 09 23:19:12.814117 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:12.814092 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04c9b7e4-6886-4d0b-abda-d0995ac30842-config-file" (OuterVolumeSpecName: "config-file") pod "04c9b7e4-6886-4d0b-abda-d0995ac30842" (UID: "04c9b7e4-6886-4d0b-abda-d0995ac30842"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:19:12.816118 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:12.816091 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c9b7e4-6886-4d0b-abda-d0995ac30842-kube-api-access-m6bsz" (OuterVolumeSpecName: "kube-api-access-m6bsz") pod "04c9b7e4-6886-4d0b-abda-d0995ac30842" (UID: "04c9b7e4-6886-4d0b-abda-d0995ac30842"). InnerVolumeSpecName "kube-api-access-m6bsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:19:12.894608 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:12.894568 2561 generic.go:356] "Generic (PLEG): container finished" podID="04c9b7e4-6886-4d0b-abda-d0995ac30842" containerID="27342608d851bff1e5fec20c57062e4d534185e969afba91100b7a2596872719" exitCode=0 Sep 09 23:19:12.895103 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:12.894617 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-fjwlz" Sep 09 23:19:12.895103 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:12.894653 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-fjwlz" event={"ID":"04c9b7e4-6886-4d0b-abda-d0995ac30842","Type":"ContainerDied","Data":"27342608d851bff1e5fec20c57062e4d534185e969afba91100b7a2596872719"} Sep 09 23:19:12.895103 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:12.894709 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-fjwlz" event={"ID":"04c9b7e4-6886-4d0b-abda-d0995ac30842","Type":"ContainerDied","Data":"f818ef3c438b1637008ea84fe3c610f27bf59f3de2c054cfe8bb68521a5a472e"} Sep 09 23:19:12.895103 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:12.894740 2561 scope.go:117] "RemoveContainer" containerID="27342608d851bff1e5fec20c57062e4d534185e969afba91100b7a2596872719" Sep 09 23:19:12.903891 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:12.903838 2561 scope.go:117] "RemoveContainer" containerID="27342608d851bff1e5fec20c57062e4d534185e969afba91100b7a2596872719" Sep 09 23:19:12.904214 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:19:12.904195 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27342608d851bff1e5fec20c57062e4d534185e969afba91100b7a2596872719\": container with ID starting with 27342608d851bff1e5fec20c57062e4d534185e969afba91100b7a2596872719 not found: ID does not exist" containerID="27342608d851bff1e5fec20c57062e4d534185e969afba91100b7a2596872719" Sep 09 23:19:12.904281 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:12.904225 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27342608d851bff1e5fec20c57062e4d534185e969afba91100b7a2596872719"} err="failed to get container status \"27342608d851bff1e5fec20c57062e4d534185e969afba91100b7a2596872719\": rpc error: code = NotFound desc = could not find container \"27342608d851bff1e5fec20c57062e4d534185e969afba91100b7a2596872719\": container with ID starting with 27342608d851bff1e5fec20c57062e4d534185e969afba91100b7a2596872719 not found: ID does not exist" Sep 09 23:19:12.914972 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:12.914944 2561 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/04c9b7e4-6886-4d0b-abda-d0995ac30842-config-file\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:19:12.914972 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:12.914971 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6bsz\" (UniqueName: \"kubernetes.io/projected/04c9b7e4-6886-4d0b-abda-d0995ac30842-kube-api-access-m6bsz\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:19:12.915187 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:12.915102 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-fjwlz"] Sep 09 23:19:12.916553 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:12.916528 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-fjwlz"] Sep 09 23:19:13.809027 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:13.808993 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04c9b7e4-6886-4d0b-abda-d0995ac30842" path="/var/lib/kubelet/pods/04c9b7e4-6886-4d0b-abda-d0995ac30842/volumes" Sep 09 23:19:17.119500 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:17.119436 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-t9k6z"] Sep 09 23:19:17.120022 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:17.119992 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04c9b7e4-6886-4d0b-abda-d0995ac30842" containerName="limitador" Sep 09 23:19:17.120022 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:17.120019 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="04c9b7e4-6886-4d0b-abda-d0995ac30842" containerName="limitador" Sep 09 23:19:17.120192 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:17.120120 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="04c9b7e4-6886-4d0b-abda-d0995ac30842" containerName="limitador" Sep 09 23:19:17.123476 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:17.123453 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t9k6z" Sep 09 23:19:17.125968 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:17.125946 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 09 23:19:17.126067 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:17.125991 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 23:19:17.126650 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:17.126634 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-bj9qt\"" Sep 09 23:19:17.126740 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:17.126720 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 09 23:19:17.131536 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:17.131289 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-t9k6z"] Sep 09 23:19:17.154085 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:17.154045 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4a837c5d-99aa-40fd-8959-d51cf2da2122-data\") pod \"postgres-69fb4594cf-t9k6z\" (UID: \"4a837c5d-99aa-40fd-8959-d51cf2da2122\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-t9k6z" Sep 09 23:19:17.154259 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:17.154113 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzb9z\" (UniqueName: \"kubernetes.io/projected/4a837c5d-99aa-40fd-8959-d51cf2da2122-kube-api-access-qzb9z\") pod \"postgres-69fb4594cf-t9k6z\" (UID: \"4a837c5d-99aa-40fd-8959-d51cf2da2122\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-t9k6z" Sep 09 23:19:17.255512 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:17.255467 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4a837c5d-99aa-40fd-8959-d51cf2da2122-data\") pod \"postgres-69fb4594cf-t9k6z\" (UID: \"4a837c5d-99aa-40fd-8959-d51cf2da2122\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-t9k6z" Sep 09 23:19:17.255696 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:17.255587 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzb9z\" (UniqueName: \"kubernetes.io/projected/4a837c5d-99aa-40fd-8959-d51cf2da2122-kube-api-access-qzb9z\") pod \"postgres-69fb4594cf-t9k6z\" (UID: \"4a837c5d-99aa-40fd-8959-d51cf2da2122\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-t9k6z" Sep 09 23:19:17.255871 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:17.255826 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4a837c5d-99aa-40fd-8959-d51cf2da2122-data\") pod \"postgres-69fb4594cf-t9k6z\" (UID: \"4a837c5d-99aa-40fd-8959-d51cf2da2122\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-t9k6z" Sep 09 23:19:17.263212 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:17.263186 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzb9z\" (UniqueName: \"kubernetes.io/projected/4a837c5d-99aa-40fd-8959-d51cf2da2122-kube-api-access-qzb9z\") pod \"postgres-69fb4594cf-t9k6z\" (UID: \"4a837c5d-99aa-40fd-8959-d51cf2da2122\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-t9k6z" Sep 09 23:19:17.436193 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:17.436159 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t9k6z" Sep 09 23:19:17.711069 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:17.710987 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-t9k6z"] Sep 09 23:19:17.919253 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:17.919216 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t9k6z" event={"ID":"4a837c5d-99aa-40fd-8959-d51cf2da2122","Type":"ContainerStarted","Data":"5db084ecfff93df77dcecf83bff54f92cd69ebfa5a5d4f0d63619374d9f191fd"} Sep 09 23:19:22.947221 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:22.947181 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t9k6z" event={"ID":"4a837c5d-99aa-40fd-8959-d51cf2da2122","Type":"ContainerStarted","Data":"8c0addabcb550accaf82f4b853a4d760d5b2af1df9f2577c4f168353afa6b7f6"} Sep 09 23:19:22.947637 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:22.947371 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t9k6z" Sep 09 23:19:22.962269 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:22.962167 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t9k6z" podStartSLOduration=0.96725196 podStartE2EDuration="5.962152081s" totalImagesPullingTime="4.994900121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:19:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:19:22.960465827 +0000 UTC m=+751.646249259" watchObservedRunningTime="2026-09-09 23:19:22.962152081 +0000 UTC m=+751.647935512" Sep 09 23:19:28.981678 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:28.981647 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t9k6z" Sep 09 23:19:36.159953 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.159915 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6bc5fdc797-46cz8"] Sep 09 23:19:36.185574 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.185505 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bc5fdc797-46cz8"] Sep 09 23:19:36.185754 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.185588 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bc5fdc797-46cz8" Sep 09 23:19:36.187980 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.187955 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 09 23:19:36.187980 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.187972 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pkl2l\"" Sep 09 23:19:36.188189 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.187955 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 09 23:19:36.188313 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.188298 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 09 23:19:36.232614 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.232576 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0abb5b29-ccba-48c0-b1f3-8ee41030965d-webhook-cert\") pod \"maas-controller-6bc5fdc797-46cz8\" (UID: \"0abb5b29-ccba-48c0-b1f3-8ee41030965d\") " pod="opendatahub/maas-controller-6bc5fdc797-46cz8" Sep 09 23:19:36.232614 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.232616 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0abb5b29-ccba-48c0-b1f3-8ee41030965d-metrics-certs\") pod \"maas-controller-6bc5fdc797-46cz8\" (UID: \"0abb5b29-ccba-48c0-b1f3-8ee41030965d\") " pod="opendatahub/maas-controller-6bc5fdc797-46cz8" Sep 09 23:19:36.232876 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.232711 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8m8p\" (UniqueName: \"kubernetes.io/projected/0abb5b29-ccba-48c0-b1f3-8ee41030965d-kube-api-access-c8m8p\") pod \"maas-controller-6bc5fdc797-46cz8\" (UID: \"0abb5b29-ccba-48c0-b1f3-8ee41030965d\") " pod="opendatahub/maas-controller-6bc5fdc797-46cz8" Sep 09 23:19:36.329514 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.329472 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-558d5b9f86-xpn66"] Sep 09 23:19:36.333579 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.333557 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-558d5b9f86-xpn66" Sep 09 23:19:36.333847 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.333822 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0abb5b29-ccba-48c0-b1f3-8ee41030965d-webhook-cert\") pod \"maas-controller-6bc5fdc797-46cz8\" (UID: \"0abb5b29-ccba-48c0-b1f3-8ee41030965d\") " pod="opendatahub/maas-controller-6bc5fdc797-46cz8" Sep 09 23:19:36.333929 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.333914 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0abb5b29-ccba-48c0-b1f3-8ee41030965d-metrics-certs\") pod \"maas-controller-6bc5fdc797-46cz8\" (UID: \"0abb5b29-ccba-48c0-b1f3-8ee41030965d\") " pod="opendatahub/maas-controller-6bc5fdc797-46cz8" Sep 09 23:19:36.333992 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.333964 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8m8p\" (UniqueName: \"kubernetes.io/projected/0abb5b29-ccba-48c0-b1f3-8ee41030965d-kube-api-access-c8m8p\") pod \"maas-controller-6bc5fdc797-46cz8\" (UID: \"0abb5b29-ccba-48c0-b1f3-8ee41030965d\") " pod="opendatahub/maas-controller-6bc5fdc797-46cz8" Sep 09 23:19:36.334043 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:19:36.334015 2561 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 09 23:19:36.334095 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:19:36.334051 2561 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Sep 09 23:19:36.334151 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:19:36.334106 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0abb5b29-ccba-48c0-b1f3-8ee41030965d-webhook-cert podName:0abb5b29-ccba-48c0-b1f3-8ee41030965d nodeName:}" failed. No retries permitted until 2026-09-09 23:19:36.834082015 +0000 UTC m=+765.519865413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/0abb5b29-ccba-48c0-b1f3-8ee41030965d-webhook-cert") pod "maas-controller-6bc5fdc797-46cz8" (UID: "0abb5b29-ccba-48c0-b1f3-8ee41030965d") : secret "maas-controller-webhook-cert" not found Sep 09 23:19:36.334151 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:19:36.334129 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0abb5b29-ccba-48c0-b1f3-8ee41030965d-metrics-certs podName:0abb5b29-ccba-48c0-b1f3-8ee41030965d nodeName:}" failed. No retries permitted until 2026-09-09 23:19:36.834118407 +0000 UTC m=+765.519901795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0abb5b29-ccba-48c0-b1f3-8ee41030965d-metrics-certs") pod "maas-controller-6bc5fdc797-46cz8" (UID: "0abb5b29-ccba-48c0-b1f3-8ee41030965d") : secret "maas-controller-metrics-tls" not found Sep 09 23:19:36.341299 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.341271 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-558d5b9f86-xpn66"] Sep 09 23:19:36.346566 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.346534 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8m8p\" (UniqueName: \"kubernetes.io/projected/0abb5b29-ccba-48c0-b1f3-8ee41030965d-kube-api-access-c8m8p\") pod \"maas-controller-6bc5fdc797-46cz8\" (UID: \"0abb5b29-ccba-48c0-b1f3-8ee41030965d\") " pod="opendatahub/maas-controller-6bc5fdc797-46cz8" Sep 09 23:19:36.434939 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.434908 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8e5f1bf-f57f-4965-bc29-43e217345fcd-webhook-cert\") pod \"maas-controller-558d5b9f86-xpn66\" (UID: \"e8e5f1bf-f57f-4965-bc29-43e217345fcd\") " pod="opendatahub/maas-controller-558d5b9f86-xpn66" Sep 09 23:19:36.435145 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.434984 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5f7r\" (UniqueName: \"kubernetes.io/projected/e8e5f1bf-f57f-4965-bc29-43e217345fcd-kube-api-access-k5f7r\") pod \"maas-controller-558d5b9f86-xpn66\" (UID: \"e8e5f1bf-f57f-4965-bc29-43e217345fcd\") " pod="opendatahub/maas-controller-558d5b9f86-xpn66" Sep 09 23:19:36.435145 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.435009 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e5f1bf-f57f-4965-bc29-43e217345fcd-metrics-certs\") pod \"maas-controller-558d5b9f86-xpn66\" (UID: \"e8e5f1bf-f57f-4965-bc29-43e217345fcd\") " pod="opendatahub/maas-controller-558d5b9f86-xpn66" Sep 09 23:19:36.535727 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.535684 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5f7r\" (UniqueName: \"kubernetes.io/projected/e8e5f1bf-f57f-4965-bc29-43e217345fcd-kube-api-access-k5f7r\") pod \"maas-controller-558d5b9f86-xpn66\" (UID: \"e8e5f1bf-f57f-4965-bc29-43e217345fcd\") " pod="opendatahub/maas-controller-558d5b9f86-xpn66" Sep 09 23:19:36.535954 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.535732 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e5f1bf-f57f-4965-bc29-43e217345fcd-metrics-certs\") pod \"maas-controller-558d5b9f86-xpn66\" (UID: \"e8e5f1bf-f57f-4965-bc29-43e217345fcd\") " pod="opendatahub/maas-controller-558d5b9f86-xpn66" Sep 09 23:19:36.535954 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.535821 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8e5f1bf-f57f-4965-bc29-43e217345fcd-webhook-cert\") pod \"maas-controller-558d5b9f86-xpn66\" (UID: \"e8e5f1bf-f57f-4965-bc29-43e217345fcd\") " pod="opendatahub/maas-controller-558d5b9f86-xpn66" Sep 09 23:19:36.536053 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:19:36.535991 2561 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 09 23:19:36.536097 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:19:36.536073 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e5f1bf-f57f-4965-bc29-43e217345fcd-webhook-cert podName:e8e5f1bf-f57f-4965-bc29-43e217345fcd nodeName:}" failed. No retries permitted until 2026-09-09 23:19:37.036053028 +0000 UTC m=+765.721836415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/e8e5f1bf-f57f-4965-bc29-43e217345fcd-webhook-cert") pod "maas-controller-558d5b9f86-xpn66" (UID: "e8e5f1bf-f57f-4965-bc29-43e217345fcd") : secret "maas-controller-webhook-cert" not found Sep 09 23:19:36.538710 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.538685 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e5f1bf-f57f-4965-bc29-43e217345fcd-metrics-certs\") pod \"maas-controller-558d5b9f86-xpn66\" (UID: \"e8e5f1bf-f57f-4965-bc29-43e217345fcd\") " pod="opendatahub/maas-controller-558d5b9f86-xpn66" Sep 09 23:19:36.545158 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.545133 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5f7r\" (UniqueName: \"kubernetes.io/projected/e8e5f1bf-f57f-4965-bc29-43e217345fcd-kube-api-access-k5f7r\") pod \"maas-controller-558d5b9f86-xpn66\" (UID: \"e8e5f1bf-f57f-4965-bc29-43e217345fcd\") " pod="opendatahub/maas-controller-558d5b9f86-xpn66" Sep 09 23:19:36.838316 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.838227 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0abb5b29-ccba-48c0-b1f3-8ee41030965d-webhook-cert\") pod \"maas-controller-6bc5fdc797-46cz8\" (UID: \"0abb5b29-ccba-48c0-b1f3-8ee41030965d\") " pod="opendatahub/maas-controller-6bc5fdc797-46cz8" Sep 09 23:19:36.838316 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.838266 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0abb5b29-ccba-48c0-b1f3-8ee41030965d-metrics-certs\") pod \"maas-controller-6bc5fdc797-46cz8\" (UID: \"0abb5b29-ccba-48c0-b1f3-8ee41030965d\") " pod="opendatahub/maas-controller-6bc5fdc797-46cz8" Sep 09 23:19:36.841049 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.841010 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0abb5b29-ccba-48c0-b1f3-8ee41030965d-metrics-certs\") pod \"maas-controller-6bc5fdc797-46cz8\" (UID: \"0abb5b29-ccba-48c0-b1f3-8ee41030965d\") " pod="opendatahub/maas-controller-6bc5fdc797-46cz8" Sep 09 23:19:36.841174 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:36.841010 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0abb5b29-ccba-48c0-b1f3-8ee41030965d-webhook-cert\") pod \"maas-controller-6bc5fdc797-46cz8\" (UID: \"0abb5b29-ccba-48c0-b1f3-8ee41030965d\") " pod="opendatahub/maas-controller-6bc5fdc797-46cz8" Sep 09 23:19:37.039753 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:37.039708 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8e5f1bf-f57f-4965-bc29-43e217345fcd-webhook-cert\") pod \"maas-controller-558d5b9f86-xpn66\" (UID: \"e8e5f1bf-f57f-4965-bc29-43e217345fcd\") " pod="opendatahub/maas-controller-558d5b9f86-xpn66" Sep 09 23:19:37.042406 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:37.042373 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8e5f1bf-f57f-4965-bc29-43e217345fcd-webhook-cert\") pod \"maas-controller-558d5b9f86-xpn66\" (UID: \"e8e5f1bf-f57f-4965-bc29-43e217345fcd\") " pod="opendatahub/maas-controller-558d5b9f86-xpn66" Sep 09 23:19:37.098182 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:37.098095 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bc5fdc797-46cz8" Sep 09 23:19:37.246227 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:37.246190 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-558d5b9f86-xpn66" Sep 09 23:19:37.370138 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:37.370113 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bc5fdc797-46cz8"] Sep 09 23:19:37.372345 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:19:37.372317 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0abb5b29_ccba_48c0_b1f3_8ee41030965d.slice/crio-74aff53cf9cccc73f28922e4c46bd55d89f6d95d29201ffaf8e7c60c776e6843 WatchSource:0}: Error finding container 74aff53cf9cccc73f28922e4c46bd55d89f6d95d29201ffaf8e7c60c776e6843: Status 404 returned error can't find the container with id 74aff53cf9cccc73f28922e4c46bd55d89f6d95d29201ffaf8e7c60c776e6843 Sep 09 23:19:37.518563 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:37.518539 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-558d5b9f86-xpn66"] Sep 09 23:19:37.521280 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:19:37.521247 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e5f1bf_f57f_4965_bc29_43e217345fcd.slice/crio-073f9c33b8669250c09c44bfbdee690a26f0076327a8dd60a8a2ca41a6489520 WatchSource:0}: Error finding container 073f9c33b8669250c09c44bfbdee690a26f0076327a8dd60a8a2ca41a6489520: Status 404 returned error can't find the container with id 073f9c33b8669250c09c44bfbdee690a26f0076327a8dd60a8a2ca41a6489520 Sep 09 23:19:38.022102 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:38.022013 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-558d5b9f86-xpn66" event={"ID":"e8e5f1bf-f57f-4965-bc29-43e217345fcd","Type":"ContainerStarted","Data":"073f9c33b8669250c09c44bfbdee690a26f0076327a8dd60a8a2ca41a6489520"} Sep 09 23:19:38.023828 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:38.023774 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bc5fdc797-46cz8" event={"ID":"0abb5b29-ccba-48c0-b1f3-8ee41030965d","Type":"ContainerStarted","Data":"74aff53cf9cccc73f28922e4c46bd55d89f6d95d29201ffaf8e7c60c776e6843"} Sep 09 23:19:41.041465 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:41.041422 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-558d5b9f86-xpn66" event={"ID":"e8e5f1bf-f57f-4965-bc29-43e217345fcd","Type":"ContainerStarted","Data":"c83f8cb0cd3a2e9d1216f5e7f5ee764de1ca03eea46b9d5cdd946029571a634b"} Sep 09 23:19:41.041996 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:41.041550 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-558d5b9f86-xpn66" Sep 09 23:19:41.043298 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:41.043273 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bc5fdc797-46cz8" event={"ID":"0abb5b29-ccba-48c0-b1f3-8ee41030965d","Type":"ContainerStarted","Data":"9ea579de9d0ce313337ef3e101ce541bcf5b0b0ba139db19d0063ab781f695e0"} Sep 09 23:19:41.043400 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:41.043323 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6bc5fdc797-46cz8" Sep 09 23:19:41.058741 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:41.058682 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-558d5b9f86-xpn66" podStartSLOduration=2.570818324 podStartE2EDuration="5.058658247s" totalImagesPullingTime="2.487839923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:19:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:19:41.057634917 +0000 UTC m=+769.743418331" watchObservedRunningTime="2026-09-09 23:19:41.058658247 +0000 UTC m=+769.744441653" Sep 09 23:19:41.073789 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:41.073727 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6bc5fdc797-46cz8" podStartSLOduration=2.443722508 podStartE2EDuration="5.073708452s" totalImagesPullingTime="2.629985944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:19:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:19:41.07192289 +0000 UTC m=+769.757706318" watchObservedRunningTime="2026-09-09 23:19:41.073708452 +0000 UTC m=+769.759491855" Sep 09 23:19:52.049648 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:52.049554 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6bc5fdc797-46cz8" Sep 09 23:19:52.050073 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:52.049656 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-558d5b9f86-xpn66" Sep 09 23:19:52.106983 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:52.106949 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6bc5fdc797-46cz8"] Sep 09 23:19:52.107193 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:52.107168 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6bc5fdc797-46cz8" podUID="0abb5b29-ccba-48c0-b1f3-8ee41030965d" containerName="manager" containerID="cri-o://9ea579de9d0ce313337ef3e101ce541bcf5b0b0ba139db19d0063ab781f695e0" gracePeriod=30 Sep 09 23:19:52.357046 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:52.357018 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bc5fdc797-46cz8" Sep 09 23:19:52.482990 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:52.482951 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0abb5b29-ccba-48c0-b1f3-8ee41030965d-metrics-certs\") pod \"0abb5b29-ccba-48c0-b1f3-8ee41030965d\" (UID: \"0abb5b29-ccba-48c0-b1f3-8ee41030965d\") " Sep 09 23:19:52.483165 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:52.483047 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8m8p\" (UniqueName: \"kubernetes.io/projected/0abb5b29-ccba-48c0-b1f3-8ee41030965d-kube-api-access-c8m8p\") pod \"0abb5b29-ccba-48c0-b1f3-8ee41030965d\" (UID: \"0abb5b29-ccba-48c0-b1f3-8ee41030965d\") " Sep 09 23:19:52.483165 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:52.483105 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0abb5b29-ccba-48c0-b1f3-8ee41030965d-webhook-cert\") pod \"0abb5b29-ccba-48c0-b1f3-8ee41030965d\" (UID: \"0abb5b29-ccba-48c0-b1f3-8ee41030965d\") " Sep 09 23:19:52.485524 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:52.485490 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0abb5b29-ccba-48c0-b1f3-8ee41030965d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0abb5b29-ccba-48c0-b1f3-8ee41030965d" (UID: "0abb5b29-ccba-48c0-b1f3-8ee41030965d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:19:52.485524 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:52.485499 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0abb5b29-ccba-48c0-b1f3-8ee41030965d-kube-api-access-c8m8p" (OuterVolumeSpecName: "kube-api-access-c8m8p") pod "0abb5b29-ccba-48c0-b1f3-8ee41030965d" (UID: "0abb5b29-ccba-48c0-b1f3-8ee41030965d"). InnerVolumeSpecName "kube-api-access-c8m8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:19:52.485524 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:52.485511 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0abb5b29-ccba-48c0-b1f3-8ee41030965d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0abb5b29-ccba-48c0-b1f3-8ee41030965d" (UID: "0abb5b29-ccba-48c0-b1f3-8ee41030965d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:19:52.584241 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:52.584192 2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0abb5b29-ccba-48c0-b1f3-8ee41030965d-metrics-certs\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:19:52.584241 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:52.584230 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8m8p\" (UniqueName: \"kubernetes.io/projected/0abb5b29-ccba-48c0-b1f3-8ee41030965d-kube-api-access-c8m8p\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:19:52.584241 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:52.584239 2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0abb5b29-ccba-48c0-b1f3-8ee41030965d-webhook-cert\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Sep 09 23:19:53.102240 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:53.102200 2561 generic.go:356] "Generic (PLEG): container finished" podID="0abb5b29-ccba-48c0-b1f3-8ee41030965d" containerID="9ea579de9d0ce313337ef3e101ce541bcf5b0b0ba139db19d0063ab781f695e0" exitCode=0 Sep 09 23:19:53.102692 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:53.102254 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bc5fdc797-46cz8" Sep 09 23:19:53.102692 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:53.102290 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bc5fdc797-46cz8" event={"ID":"0abb5b29-ccba-48c0-b1f3-8ee41030965d","Type":"ContainerDied","Data":"9ea579de9d0ce313337ef3e101ce541bcf5b0b0ba139db19d0063ab781f695e0"} Sep 09 23:19:53.102692 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:53.102345 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bc5fdc797-46cz8" event={"ID":"0abb5b29-ccba-48c0-b1f3-8ee41030965d","Type":"ContainerDied","Data":"74aff53cf9cccc73f28922e4c46bd55d89f6d95d29201ffaf8e7c60c776e6843"} Sep 09 23:19:53.102692 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:53.102367 2561 scope.go:117] "RemoveContainer" containerID="9ea579de9d0ce313337ef3e101ce541bcf5b0b0ba139db19d0063ab781f695e0" Sep 09 23:19:53.111582 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:53.111558 2561 scope.go:117] "RemoveContainer" containerID="9ea579de9d0ce313337ef3e101ce541bcf5b0b0ba139db19d0063ab781f695e0" Sep 09 23:19:53.111906 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:19:53.111883 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea579de9d0ce313337ef3e101ce541bcf5b0b0ba139db19d0063ab781f695e0\": container with ID starting with 9ea579de9d0ce313337ef3e101ce541bcf5b0b0ba139db19d0063ab781f695e0 not found: ID does not exist" containerID="9ea579de9d0ce313337ef3e101ce541bcf5b0b0ba139db19d0063ab781f695e0" Sep 09 23:19:53.111999 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:53.111915 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea579de9d0ce313337ef3e101ce541bcf5b0b0ba139db19d0063ab781f695e0"} err="failed to get container status \"9ea579de9d0ce313337ef3e101ce541bcf5b0b0ba139db19d0063ab781f695e0\": rpc error: code = NotFound desc = could not find container \"9ea579de9d0ce313337ef3e101ce541bcf5b0b0ba139db19d0063ab781f695e0\": container with ID starting with 9ea579de9d0ce313337ef3e101ce541bcf5b0b0ba139db19d0063ab781f695e0 not found: ID does not exist" Sep 09 23:19:53.122198 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:53.122132 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6bc5fdc797-46cz8"] Sep 09 23:19:53.123880 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:53.123837 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6bc5fdc797-46cz8"] Sep 09 23:19:53.809755 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:19:53.809717 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0abb5b29-ccba-48c0-b1f3-8ee41030965d" path="/var/lib/kubelet/pods/0abb5b29-ccba-48c0-b1f3-8ee41030965d/volumes" Sep 09 23:20:14.401480 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.401389 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f576956d9-mzv7x"] Sep 09 23:20:14.401958 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.401770 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0abb5b29-ccba-48c0-b1f3-8ee41030965d" containerName="manager" Sep 09 23:20:14.401958 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.401784 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0abb5b29-ccba-48c0-b1f3-8ee41030965d" containerName="manager" Sep 09 23:20:14.401958 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.401847 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0abb5b29-ccba-48c0-b1f3-8ee41030965d" containerName="manager" Sep 09 23:20:14.408694 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.408662 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f576956d9-mzv7x" Sep 09 23:20:14.412054 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.412030 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 09 23:20:14.412220 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.412054 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9flwz\"" Sep 09 23:20:14.412220 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.412054 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 09 23:20:14.419583 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.419559 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f576956d9-mzv7x"] Sep 09 23:20:14.486363 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.486326 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s58dx\" (UniqueName: \"kubernetes.io/projected/8dc3549a-cfbd-4890-b4c2-78394425a147-kube-api-access-s58dx\") pod \"maas-api-f576956d9-mzv7x\" (UID: \"8dc3549a-cfbd-4890-b4c2-78394425a147\") " pod="odh-ai-gateway-infra/maas-api-f576956d9-mzv7x" Sep 09 23:20:14.486549 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.486385 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dc3549a-cfbd-4890-b4c2-78394425a147-metrics-certs\") pod \"maas-api-f576956d9-mzv7x\" (UID: \"8dc3549a-cfbd-4890-b4c2-78394425a147\") " pod="odh-ai-gateway-infra/maas-api-f576956d9-mzv7x" Sep 09 23:20:14.486549 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.486446 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dc3549a-cfbd-4890-b4c2-78394425a147-maas-api-tls\") pod \"maas-api-f576956d9-mzv7x\" (UID: \"8dc3549a-cfbd-4890-b4c2-78394425a147\") " pod="odh-ai-gateway-infra/maas-api-f576956d9-mzv7x" Sep 09 23:20:14.486549 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.486498 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc3549a-cfbd-4890-b4c2-78394425a147-openshift-service-ca\") pod \"maas-api-f576956d9-mzv7x\" (UID: \"8dc3549a-cfbd-4890-b4c2-78394425a147\") " pod="odh-ai-gateway-infra/maas-api-f576956d9-mzv7x" Sep 09 23:20:14.587989 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.587948 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dc3549a-cfbd-4890-b4c2-78394425a147-metrics-certs\") pod \"maas-api-f576956d9-mzv7x\" (UID: \"8dc3549a-cfbd-4890-b4c2-78394425a147\") " pod="odh-ai-gateway-infra/maas-api-f576956d9-mzv7x" Sep 09 23:20:14.587989 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.587990 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dc3549a-cfbd-4890-b4c2-78394425a147-maas-api-tls\") pod \"maas-api-f576956d9-mzv7x\" (UID: \"8dc3549a-cfbd-4890-b4c2-78394425a147\") " pod="odh-ai-gateway-infra/maas-api-f576956d9-mzv7x" Sep 09 23:20:14.588214 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.588010 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc3549a-cfbd-4890-b4c2-78394425a147-openshift-service-ca\") pod \"maas-api-f576956d9-mzv7x\" (UID: \"8dc3549a-cfbd-4890-b4c2-78394425a147\") " pod="odh-ai-gateway-infra/maas-api-f576956d9-mzv7x" Sep 09 23:20:14.588214 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.588071 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s58dx\" (UniqueName: \"kubernetes.io/projected/8dc3549a-cfbd-4890-b4c2-78394425a147-kube-api-access-s58dx\") pod \"maas-api-f576956d9-mzv7x\" (UID: \"8dc3549a-cfbd-4890-b4c2-78394425a147\") " pod="odh-ai-gateway-infra/maas-api-f576956d9-mzv7x" Sep 09 23:20:14.588888 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.588747 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dc3549a-cfbd-4890-b4c2-78394425a147-openshift-service-ca\") pod \"maas-api-f576956d9-mzv7x\" (UID: \"8dc3549a-cfbd-4890-b4c2-78394425a147\") " pod="odh-ai-gateway-infra/maas-api-f576956d9-mzv7x" Sep 09 23:20:14.590707 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.590684 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dc3549a-cfbd-4890-b4c2-78394425a147-maas-api-tls\") pod \"maas-api-f576956d9-mzv7x\" (UID: \"8dc3549a-cfbd-4890-b4c2-78394425a147\") " pod="odh-ai-gateway-infra/maas-api-f576956d9-mzv7x" Sep 09 23:20:14.590815 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.590734 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dc3549a-cfbd-4890-b4c2-78394425a147-metrics-certs\") pod \"maas-api-f576956d9-mzv7x\" (UID: \"8dc3549a-cfbd-4890-b4c2-78394425a147\") " pod="odh-ai-gateway-infra/maas-api-f576956d9-mzv7x" Sep 09 23:20:14.596085 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.596062 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s58dx\" (UniqueName: \"kubernetes.io/projected/8dc3549a-cfbd-4890-b4c2-78394425a147-kube-api-access-s58dx\") pod \"maas-api-f576956d9-mzv7x\" (UID: \"8dc3549a-cfbd-4890-b4c2-78394425a147\") " pod="odh-ai-gateway-infra/maas-api-f576956d9-mzv7x" Sep 09 23:20:14.720770 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.720728 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f576956d9-mzv7x" Sep 09 23:20:14.908742 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.908701 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-h555r"] Sep 09 23:20:14.912407 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.912383 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-h555r" Sep 09 23:20:14.914733 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.914705 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-46264\"" Sep 09 23:20:14.914906 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.914738 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 09 23:20:14.923080 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.923050 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-h555r"] Sep 09 23:20:14.992323 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.992223 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8317f8-8b9c-4309-94da-ae5b484e14ab-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-h555r\" (UID: \"eb8317f8-8b9c-4309-94da-ae5b484e14ab\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-h555r" Sep 09 23:20:14.992323 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.992277 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf6tm\" (UniqueName: \"kubernetes.io/projected/eb8317f8-8b9c-4309-94da-ae5b484e14ab-kube-api-access-qf6tm\") pod \"payload-pre-processing-59fbc47bf8-h555r\" (UID: \"eb8317f8-8b9c-4309-94da-ae5b484e14ab\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-h555r" Sep 09 23:20:14.997936 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:14.997911 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f576956d9-mzv7x"] Sep 09 23:20:15.000577 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:20:15.000551 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc3549a_cfbd_4890_b4c2_78394425a147.slice/crio-da56316989145b02a24800a9c22447cc399fc932ce4a2b5b1296a01912ce6a7e WatchSource:0}: Error finding container da56316989145b02a24800a9c22447cc399fc932ce4a2b5b1296a01912ce6a7e: Status 404 returned error can't find the container with id da56316989145b02a24800a9c22447cc399fc932ce4a2b5b1296a01912ce6a7e Sep 09 23:20:15.093593 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:15.093555 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8317f8-8b9c-4309-94da-ae5b484e14ab-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-h555r\" (UID: \"eb8317f8-8b9c-4309-94da-ae5b484e14ab\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-h555r" Sep 09 23:20:15.093792 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:15.093601 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf6tm\" (UniqueName: \"kubernetes.io/projected/eb8317f8-8b9c-4309-94da-ae5b484e14ab-kube-api-access-qf6tm\") pod \"payload-pre-processing-59fbc47bf8-h555r\" (UID: \"eb8317f8-8b9c-4309-94da-ae5b484e14ab\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-h555r" Sep 09 23:20:15.094548 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:15.094522 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8317f8-8b9c-4309-94da-ae5b484e14ab-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-h555r\" (UID: \"eb8317f8-8b9c-4309-94da-ae5b484e14ab\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-h555r" Sep 09 23:20:15.103815 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:15.103776 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf6tm\" (UniqueName: \"kubernetes.io/projected/eb8317f8-8b9c-4309-94da-ae5b484e14ab-kube-api-access-qf6tm\") pod \"payload-pre-processing-59fbc47bf8-h555r\" (UID: \"eb8317f8-8b9c-4309-94da-ae5b484e14ab\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-h555r" Sep 09 23:20:15.205226 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:15.205189 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f576956d9-mzv7x" event={"ID":"8dc3549a-cfbd-4890-b4c2-78394425a147","Type":"ContainerStarted","Data":"da56316989145b02a24800a9c22447cc399fc932ce4a2b5b1296a01912ce6a7e"} Sep 09 23:20:15.224472 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:15.224436 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-h555r" Sep 09 23:20:15.496332 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:15.496303 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-h555r"] Sep 09 23:20:15.498634 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:20:15.498599 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8317f8_8b9c_4309_94da_ae5b484e14ab.slice/crio-6fe122e74c17256dc9885ab2dcbf9776fc3c2e97cdfcf81e73eef170001f31fc WatchSource:0}: Error finding container 6fe122e74c17256dc9885ab2dcbf9776fc3c2e97cdfcf81e73eef170001f31fc: Status 404 returned error can't find the container with id 6fe122e74c17256dc9885ab2dcbf9776fc3c2e97cdfcf81e73eef170001f31fc Sep 09 23:20:16.213328 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:16.213275 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-h555r" event={"ID":"eb8317f8-8b9c-4309-94da-ae5b484e14ab","Type":"ContainerStarted","Data":"6fe122e74c17256dc9885ab2dcbf9776fc3c2e97cdfcf81e73eef170001f31fc"} Sep 09 23:20:17.221573 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:17.221522 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f576956d9-mzv7x" event={"ID":"8dc3549a-cfbd-4890-b4c2-78394425a147","Type":"ContainerStarted","Data":"95f9643c72c1a3cd1623168d05544530f33ac72c767c3ae295cc771e583c8f95"} Sep 09 23:20:17.222047 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:17.221633 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f576956d9-mzv7x" Sep 09 23:20:17.239071 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:17.239007 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f576956d9-mzv7x" podStartSLOduration=1.427952027 podStartE2EDuration="3.238980075s" totalImagesPullingTime="1.811028048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:20:17.236603986 +0000 UTC m=+805.922387434" watchObservedRunningTime="2026-09-09 23:20:17.238980075 +0000 UTC m=+805.924763479" Sep 09 23:20:18.227925 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:18.227884 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-h555r" event={"ID":"eb8317f8-8b9c-4309-94da-ae5b484e14ab","Type":"ContainerStarted","Data":"4d21e42fc05d3ea8b841292ab874627ec532bf4d456728db338c79b1a738df4c"} Sep 09 23:20:18.242654 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:18.242602 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-h555r" podStartSLOduration=1.710325114 podStartE2EDuration="4.242584417s" totalImagesPullingTime="2.532259303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:20:18.241761989 +0000 UTC m=+806.927545623" watchObservedRunningTime="2026-09-09 23:20:18.242584417 +0000 UTC m=+806.928367832" Sep 09 23:20:19.234109 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:19.234046 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-h555r" Sep 09 23:20:23.233711 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:23.233674 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f576956d9-mzv7x" Sep 09 23:20:30.240681 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:30.240648 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-h555r" Sep 09 23:20:30.800703 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:30.800649 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-859856b874-678lf"] Sep 09 23:20:30.805938 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:30.805911 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:20:30.808255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:30.808225 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-sbz4r\"" Sep 09 23:20:30.808255 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:30.808244 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 09 23:20:30.808467 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:30.808237 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 09 23:20:30.808870 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:30.808832 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 09 23:20:30.814610 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:30.814572 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-859856b874-678lf"] Sep 09 23:20:30.958274 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:30.958242 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj8cn\" (UniqueName: \"kubernetes.io/projected/ecff2538-a36b-41ca-8484-c5f54bb2cc89-kube-api-access-dj8cn\") pod \"e2e-distinct-2-simulated-kserve-859856b874-678lf\" (UID: \"ecff2538-a36b-41ca-8484-c5f54bb2cc89\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:20:30.958439 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:30.958316 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecff2538-a36b-41ca-8484-c5f54bb2cc89-home\") pod \"e2e-distinct-2-simulated-kserve-859856b874-678lf\" (UID: \"ecff2538-a36b-41ca-8484-c5f54bb2cc89\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:20:30.958439 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:30.958342 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecff2538-a36b-41ca-8484-c5f54bb2cc89-model-cache\") pod \"e2e-distinct-2-simulated-kserve-859856b874-678lf\" (UID: \"ecff2538-a36b-41ca-8484-c5f54bb2cc89\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:20:30.958439 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:30.958362 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecff2538-a36b-41ca-8484-c5f54bb2cc89-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-859856b874-678lf\" (UID: \"ecff2538-a36b-41ca-8484-c5f54bb2cc89\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:20:30.958439 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:30.958378 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecff2538-a36b-41ca-8484-c5f54bb2cc89-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-859856b874-678lf\" (UID: \"ecff2538-a36b-41ca-8484-c5f54bb2cc89\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:20:30.958575 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:30.958483 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecff2538-a36b-41ca-8484-c5f54bb2cc89-dshm\") pod \"e2e-distinct-2-simulated-kserve-859856b874-678lf\" (UID: \"ecff2538-a36b-41ca-8484-c5f54bb2cc89\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:20:31.059339 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:31.059233 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecff2538-a36b-41ca-8484-c5f54bb2cc89-dshm\") pod \"e2e-distinct-2-simulated-kserve-859856b874-678lf\" (UID: \"ecff2538-a36b-41ca-8484-c5f54bb2cc89\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:20:31.059339 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:31.059298 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj8cn\" (UniqueName: \"kubernetes.io/projected/ecff2538-a36b-41ca-8484-c5f54bb2cc89-kube-api-access-dj8cn\") pod \"e2e-distinct-2-simulated-kserve-859856b874-678lf\" (UID: \"ecff2538-a36b-41ca-8484-c5f54bb2cc89\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:20:31.059641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:31.059346 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecff2538-a36b-41ca-8484-c5f54bb2cc89-home\") pod \"e2e-distinct-2-simulated-kserve-859856b874-678lf\" (UID: \"ecff2538-a36b-41ca-8484-c5f54bb2cc89\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:20:31.059641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:31.059362 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecff2538-a36b-41ca-8484-c5f54bb2cc89-model-cache\") pod \"e2e-distinct-2-simulated-kserve-859856b874-678lf\" (UID: \"ecff2538-a36b-41ca-8484-c5f54bb2cc89\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:20:31.059641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:31.059384 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecff2538-a36b-41ca-8484-c5f54bb2cc89-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-859856b874-678lf\" (UID: \"ecff2538-a36b-41ca-8484-c5f54bb2cc89\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:20:31.059641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:31.059398 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecff2538-a36b-41ca-8484-c5f54bb2cc89-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-859856b874-678lf\" (UID: \"ecff2538-a36b-41ca-8484-c5f54bb2cc89\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:20:31.059888 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:31.059782 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecff2538-a36b-41ca-8484-c5f54bb2cc89-home\") pod \"e2e-distinct-2-simulated-kserve-859856b874-678lf\" (UID: \"ecff2538-a36b-41ca-8484-c5f54bb2cc89\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:20:31.059888 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:31.059829 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecff2538-a36b-41ca-8484-c5f54bb2cc89-model-cache\") pod \"e2e-distinct-2-simulated-kserve-859856b874-678lf\" (UID: \"ecff2538-a36b-41ca-8484-c5f54bb2cc89\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:20:31.059888 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:31.059838 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecff2538-a36b-41ca-8484-c5f54bb2cc89-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-859856b874-678lf\" (UID: \"ecff2538-a36b-41ca-8484-c5f54bb2cc89\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:20:31.061713 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:31.061684 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecff2538-a36b-41ca-8484-c5f54bb2cc89-dshm\") pod \"e2e-distinct-2-simulated-kserve-859856b874-678lf\" (UID: \"ecff2538-a36b-41ca-8484-c5f54bb2cc89\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:20:31.062011 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:31.061993 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecff2538-a36b-41ca-8484-c5f54bb2cc89-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-859856b874-678lf\" (UID: \"ecff2538-a36b-41ca-8484-c5f54bb2cc89\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:20:31.067118 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:31.067098 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj8cn\" (UniqueName: \"kubernetes.io/projected/ecff2538-a36b-41ca-8484-c5f54bb2cc89-kube-api-access-dj8cn\") pod \"e2e-distinct-2-simulated-kserve-859856b874-678lf\" (UID: \"ecff2538-a36b-41ca-8484-c5f54bb2cc89\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:20:31.121332 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:31.121284 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:20:31.413916 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:31.413888 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-859856b874-678lf"] Sep 09 23:20:31.416262 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:20:31.416234 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecff2538_a36b_41ca_8484_c5f54bb2cc89.slice/crio-14fddd58419f51652a823f99a6e365d6ab1c7a923ff5296f8401c285778f4899 WatchSource:0}: Error finding container 14fddd58419f51652a823f99a6e365d6ab1c7a923ff5296f8401c285778f4899: Status 404 returned error can't find the container with id 14fddd58419f51652a823f99a6e365d6ab1c7a923ff5296f8401c285778f4899 Sep 09 23:20:32.298068 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:32.298028 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" event={"ID":"ecff2538-a36b-41ca-8484-c5f54bb2cc89","Type":"ContainerStarted","Data":"14fddd58419f51652a823f99a6e365d6ab1c7a923ff5296f8401c285778f4899"} Sep 09 23:20:33.304753 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:33.304716 2561 generic.go:356] "Generic (PLEG): container finished" podID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" containerID="c2e72d5eb0d4c9a9798cf4458539f06bca91473e4f54bd908d31e193f3f98dbf" exitCode=0 Sep 09 23:20:33.305267 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:33.304772 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" event={"ID":"ecff2538-a36b-41ca-8484-c5f54bb2cc89","Type":"ContainerDied","Data":"c2e72d5eb0d4c9a9798cf4458539f06bca91473e4f54bd908d31e193f3f98dbf"} Sep 09 23:20:33.305267 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:33.305171 2561 scope.go:117] "RemoveContainer" containerID="c2e72d5eb0d4c9a9798cf4458539f06bca91473e4f54bd908d31e193f3f98dbf" Sep 09 23:20:34.311822 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:34.311790 2561 generic.go:356] "Generic (PLEG): container finished" podID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" containerID="2b454d366bb729a828d73f395c49fd8e4fef46e9ee1b7ee2fd4a40477429198f" exitCode=0 Sep 09 23:20:34.312322 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:34.311887 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" event={"ID":"ecff2538-a36b-41ca-8484-c5f54bb2cc89","Type":"ContainerDied","Data":"2b454d366bb729a828d73f395c49fd8e4fef46e9ee1b7ee2fd4a40477429198f"} Sep 09 23:20:34.312322 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:34.311967 2561 scope.go:117] "RemoveContainer" containerID="c2e72d5eb0d4c9a9798cf4458539f06bca91473e4f54bd908d31e193f3f98dbf" Sep 09 23:20:34.312322 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:34.312263 2561 scope.go:117] "RemoveContainer" containerID="2b454d366bb729a828d73f395c49fd8e4fef46e9ee1b7ee2fd4a40477429198f" Sep 09 23:20:34.312493 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:20:34.312476 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:20:35.319339 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:35.319306 2561 scope.go:117] "RemoveContainer" containerID="2b454d366bb729a828d73f395c49fd8e4fef46e9ee1b7ee2fd4a40477429198f" Sep 09 23:20:35.319764 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:20:35.319491 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:20:35.691324 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:35.691290 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl"] Sep 09 23:20:35.696395 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:35.696371 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:20:35.698727 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:35.698699 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 09 23:20:35.708629 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:35.708602 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl"] Sep 09 23:20:35.806600 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:35.806572 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc27533-85d7-4f55-8f45-dcd10685b8f6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-6685d86975-8tlsl\" (UID: \"bbc27533-85d7-4f55-8f45-dcd10685b8f6\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:20:35.806600 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:35.806605 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbc27533-85d7-4f55-8f45-dcd10685b8f6-home\") pod \"e2e-distinct-simulated-kserve-6685d86975-8tlsl\" (UID: \"bbc27533-85d7-4f55-8f45-dcd10685b8f6\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:20:35.806915 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:35.806625 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr9lm\" (UniqueName: \"kubernetes.io/projected/bbc27533-85d7-4f55-8f45-dcd10685b8f6-kube-api-access-cr9lm\") pod \"e2e-distinct-simulated-kserve-6685d86975-8tlsl\" (UID: \"bbc27533-85d7-4f55-8f45-dcd10685b8f6\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:20:35.806915 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:35.806682 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc27533-85d7-4f55-8f45-dcd10685b8f6-dshm\") pod \"e2e-distinct-simulated-kserve-6685d86975-8tlsl\" (UID: \"bbc27533-85d7-4f55-8f45-dcd10685b8f6\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:20:35.806915 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:35.806709 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc27533-85d7-4f55-8f45-dcd10685b8f6-model-cache\") pod \"e2e-distinct-simulated-kserve-6685d86975-8tlsl\" (UID: \"bbc27533-85d7-4f55-8f45-dcd10685b8f6\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:20:35.806915 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:35.806726 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc27533-85d7-4f55-8f45-dcd10685b8f6-tls-certs\") pod \"e2e-distinct-simulated-kserve-6685d86975-8tlsl\" (UID: \"bbc27533-85d7-4f55-8f45-dcd10685b8f6\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:20:35.907840 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:35.907797 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc27533-85d7-4f55-8f45-dcd10685b8f6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-6685d86975-8tlsl\" (UID: \"bbc27533-85d7-4f55-8f45-dcd10685b8f6\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:20:35.907840 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:35.907875 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbc27533-85d7-4f55-8f45-dcd10685b8f6-home\") pod \"e2e-distinct-simulated-kserve-6685d86975-8tlsl\" (UID: \"bbc27533-85d7-4f55-8f45-dcd10685b8f6\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:20:35.908106 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:35.908092 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr9lm\" (UniqueName: \"kubernetes.io/projected/bbc27533-85d7-4f55-8f45-dcd10685b8f6-kube-api-access-cr9lm\") pod \"e2e-distinct-simulated-kserve-6685d86975-8tlsl\" (UID: \"bbc27533-85d7-4f55-8f45-dcd10685b8f6\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:20:35.908163 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:35.908141 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc27533-85d7-4f55-8f45-dcd10685b8f6-dshm\") pod \"e2e-distinct-simulated-kserve-6685d86975-8tlsl\" (UID: \"bbc27533-85d7-4f55-8f45-dcd10685b8f6\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:20:35.908216 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:35.908191 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc27533-85d7-4f55-8f45-dcd10685b8f6-model-cache\") pod \"e2e-distinct-simulated-kserve-6685d86975-8tlsl\" (UID: \"bbc27533-85d7-4f55-8f45-dcd10685b8f6\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:20:35.908265 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:35.908214 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc27533-85d7-4f55-8f45-dcd10685b8f6-tls-certs\") pod \"e2e-distinct-simulated-kserve-6685d86975-8tlsl\" (UID: \"bbc27533-85d7-4f55-8f45-dcd10685b8f6\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:20:35.908315 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:35.908270 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc27533-85d7-4f55-8f45-dcd10685b8f6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-6685d86975-8tlsl\" (UID: \"bbc27533-85d7-4f55-8f45-dcd10685b8f6\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:20:35.908792 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:35.908766 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbc27533-85d7-4f55-8f45-dcd10685b8f6-home\") pod \"e2e-distinct-simulated-kserve-6685d86975-8tlsl\" (UID: \"bbc27533-85d7-4f55-8f45-dcd10685b8f6\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:20:35.908936 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:35.908792 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc27533-85d7-4f55-8f45-dcd10685b8f6-model-cache\") pod \"e2e-distinct-simulated-kserve-6685d86975-8tlsl\" (UID: \"bbc27533-85d7-4f55-8f45-dcd10685b8f6\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:20:35.910647 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:35.910626 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc27533-85d7-4f55-8f45-dcd10685b8f6-dshm\") pod \"e2e-distinct-simulated-kserve-6685d86975-8tlsl\" (UID: \"bbc27533-85d7-4f55-8f45-dcd10685b8f6\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:20:35.910932 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:35.910914 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc27533-85d7-4f55-8f45-dcd10685b8f6-tls-certs\") pod \"e2e-distinct-simulated-kserve-6685d86975-8tlsl\" (UID: \"bbc27533-85d7-4f55-8f45-dcd10685b8f6\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:20:35.915812 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:35.915785 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr9lm\" (UniqueName: \"kubernetes.io/projected/bbc27533-85d7-4f55-8f45-dcd10685b8f6-kube-api-access-cr9lm\") pod \"e2e-distinct-simulated-kserve-6685d86975-8tlsl\" (UID: \"bbc27533-85d7-4f55-8f45-dcd10685b8f6\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:20:36.010257 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:36.010160 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:20:36.282523 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:36.282496 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl"] Sep 09 23:20:36.284899 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:20:36.284846 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc27533_85d7_4f55_8f45_dcd10685b8f6.slice/crio-880d8a96f652ab61edb52dc66da611919428636830d8362021ffd40b4a98a8b3 WatchSource:0}: Error finding container 880d8a96f652ab61edb52dc66da611919428636830d8362021ffd40b4a98a8b3: Status 404 returned error can't find the container with id 880d8a96f652ab61edb52dc66da611919428636830d8362021ffd40b4a98a8b3 Sep 09 23:20:36.326005 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:36.325967 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" event={"ID":"bbc27533-85d7-4f55-8f45-dcd10685b8f6","Type":"ContainerStarted","Data":"880d8a96f652ab61edb52dc66da611919428636830d8362021ffd40b4a98a8b3"} Sep 09 23:20:37.332575 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:37.332505 2561 generic.go:356] "Generic (PLEG): container finished" podID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" containerID="6dd7716d95b032dd54d10b89b90d23842307583097a83cae74b0871f4541a014" exitCode=0 Sep 09 23:20:37.333008 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:37.332605 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" event={"ID":"bbc27533-85d7-4f55-8f45-dcd10685b8f6","Type":"ContainerDied","Data":"6dd7716d95b032dd54d10b89b90d23842307583097a83cae74b0871f4541a014"} Sep 09 23:20:37.333008 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:37.332875 2561 scope.go:117] "RemoveContainer" containerID="6dd7716d95b032dd54d10b89b90d23842307583097a83cae74b0871f4541a014" Sep 09 23:20:38.339755 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:38.339714 2561 generic.go:356] "Generic (PLEG): container finished" podID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" containerID="957bba002c5d74a7720d8c752fb68b00124d4f04c64eac3963e154aea4068010" exitCode=0 Sep 09 23:20:38.340269 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:38.339770 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" event={"ID":"bbc27533-85d7-4f55-8f45-dcd10685b8f6","Type":"ContainerDied","Data":"957bba002c5d74a7720d8c752fb68b00124d4f04c64eac3963e154aea4068010"} Sep 09 23:20:38.340269 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:38.339833 2561 scope.go:117] "RemoveContainer" containerID="6dd7716d95b032dd54d10b89b90d23842307583097a83cae74b0871f4541a014" Sep 09 23:20:38.340269 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:38.340135 2561 scope.go:117] "RemoveContainer" containerID="957bba002c5d74a7720d8c752fb68b00124d4f04c64eac3963e154aea4068010" Sep 09 23:20:38.340401 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:20:38.340380 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:20:39.346705 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:39.346677 2561 scope.go:117] "RemoveContainer" containerID="957bba002c5d74a7720d8c752fb68b00124d4f04c64eac3963e154aea4068010" Sep 09 23:20:39.347140 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:20:39.346871 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:20:41.121591 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:41.121554 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:20:41.122064 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:41.121630 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:20:41.122064 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:41.121653 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:20:41.122064 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:41.121674 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:20:41.122180 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:41.122161 2561 scope.go:117] "RemoveContainer" containerID="2b454d366bb729a828d73f395c49fd8e4fef46e9ee1b7ee2fd4a40477429198f" Sep 09 23:20:41.122383 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:20:41.122362 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:20:46.010501 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:46.010456 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:20:46.010956 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:46.010529 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:20:46.010956 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:46.010544 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:20:46.010956 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:46.010557 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:20:46.011060 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:46.010976 2561 scope.go:117] "RemoveContainer" containerID="957bba002c5d74a7720d8c752fb68b00124d4f04c64eac3963e154aea4068010" Sep 09 23:20:46.011166 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:20:46.011147 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:20:54.804550 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:54.804515 2561 scope.go:117] "RemoveContainer" containerID="2b454d366bb729a828d73f395c49fd8e4fef46e9ee1b7ee2fd4a40477429198f" Sep 09 23:20:55.430549 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:55.430518 2561 generic.go:356] "Generic (PLEG): container finished" podID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" containerID="4e40b67076ea9b44e79ca5fca71798713823a8f009e5fc9147873cbf90b1433e" exitCode=0 Sep 09 23:20:55.430727 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:55.430587 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" event={"ID":"ecff2538-a36b-41ca-8484-c5f54bb2cc89","Type":"ContainerDied","Data":"4e40b67076ea9b44e79ca5fca71798713823a8f009e5fc9147873cbf90b1433e"} Sep 09 23:20:55.430727 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:55.430646 2561 scope.go:117] "RemoveContainer" containerID="2b454d366bb729a828d73f395c49fd8e4fef46e9ee1b7ee2fd4a40477429198f" Sep 09 23:20:55.431058 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:55.431042 2561 scope.go:117] "RemoveContainer" containerID="4e40b67076ea9b44e79ca5fca71798713823a8f009e5fc9147873cbf90b1433e" Sep 09 23:20:55.431271 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:20:55.431251 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:20:58.804222 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:58.804184 2561 scope.go:117] "RemoveContainer" containerID="957bba002c5d74a7720d8c752fb68b00124d4f04c64eac3963e154aea4068010" Sep 09 23:20:59.453539 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:59.453497 2561 generic.go:356] "Generic (PLEG): container finished" podID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" containerID="81027071c0141a28697faa6ff3c3c1b521ba827d992a1649488d26439ec13748" exitCode=0 Sep 09 23:20:59.453708 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:59.453563 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" event={"ID":"bbc27533-85d7-4f55-8f45-dcd10685b8f6","Type":"ContainerDied","Data":"81027071c0141a28697faa6ff3c3c1b521ba827d992a1649488d26439ec13748"} Sep 09 23:20:59.453708 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:59.453617 2561 scope.go:117] "RemoveContainer" containerID="957bba002c5d74a7720d8c752fb68b00124d4f04c64eac3963e154aea4068010" Sep 09 23:20:59.454022 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:20:59.454004 2561 scope.go:117] "RemoveContainer" containerID="81027071c0141a28697faa6ff3c3c1b521ba827d992a1649488d26439ec13748" Sep 09 23:20:59.454227 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:20:59.454202 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:21:01.121882 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:01.121820 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:21:01.122285 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:01.121907 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:21:01.122285 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:01.121923 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:21:01.122285 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:01.121937 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:21:01.122406 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:01.122308 2561 scope.go:117] "RemoveContainer" containerID="4e40b67076ea9b44e79ca5fca71798713823a8f009e5fc9147873cbf90b1433e" Sep 09 23:21:01.122498 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:21:01.122479 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:21:01.465824 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:01.465801 2561 scope.go:117] "RemoveContainer" containerID="4e40b67076ea9b44e79ca5fca71798713823a8f009e5fc9147873cbf90b1433e" Sep 09 23:21:01.466014 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:21:01.466001 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:21:06.010469 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:06.010428 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:21:06.010900 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:06.010500 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:21:06.010900 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:06.010522 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:21:06.010900 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:06.010537 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:21:06.011010 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:06.010996 2561 scope.go:117] "RemoveContainer" containerID="81027071c0141a28697faa6ff3c3c1b521ba827d992a1649488d26439ec13748" Sep 09 23:21:06.011267 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:21:06.011242 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:21:15.804321 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:15.804284 2561 scope.go:117] "RemoveContainer" containerID="4e40b67076ea9b44e79ca5fca71798713823a8f009e5fc9147873cbf90b1433e" Sep 09 23:21:15.804777 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:15.804686 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 23:21:16.542809 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:16.542776 2561 generic.go:356] "Generic (PLEG): container finished" podID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" containerID="a2980bece3135c5fa297152bb0a2f81aa29d538e94e79026a9d33e90db2103d8" exitCode=0 Sep 09 23:21:16.542995 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:16.542832 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" event={"ID":"ecff2538-a36b-41ca-8484-c5f54bb2cc89","Type":"ContainerDied","Data":"a2980bece3135c5fa297152bb0a2f81aa29d538e94e79026a9d33e90db2103d8"} Sep 09 23:21:16.542995 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:16.542910 2561 scope.go:117] "RemoveContainer" containerID="4e40b67076ea9b44e79ca5fca71798713823a8f009e5fc9147873cbf90b1433e" Sep 09 23:21:16.543301 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:16.543286 2561 scope.go:117] "RemoveContainer" containerID="a2980bece3135c5fa297152bb0a2f81aa29d538e94e79026a9d33e90db2103d8" Sep 09 23:21:16.543484 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:21:16.543465 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:21:19.804528 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:19.804495 2561 scope.go:117] "RemoveContainer" containerID="81027071c0141a28697faa6ff3c3c1b521ba827d992a1649488d26439ec13748" Sep 09 23:21:20.567698 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:20.567669 2561 generic.go:356] "Generic (PLEG): container finished" podID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" containerID="a30728a8c607be25a1fe9027ab4ae1d4c22addd6a6bafc8f1db68139296cdbd0" exitCode=0 Sep 09 23:21:20.567889 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:20.567745 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" event={"ID":"bbc27533-85d7-4f55-8f45-dcd10685b8f6","Type":"ContainerDied","Data":"a30728a8c607be25a1fe9027ab4ae1d4c22addd6a6bafc8f1db68139296cdbd0"} Sep 09 23:21:20.567889 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:20.567804 2561 scope.go:117] "RemoveContainer" containerID="81027071c0141a28697faa6ff3c3c1b521ba827d992a1649488d26439ec13748" Sep 09 23:21:20.568218 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:20.568205 2561 scope.go:117] "RemoveContainer" containerID="a30728a8c607be25a1fe9027ab4ae1d4c22addd6a6bafc8f1db68139296cdbd0" Sep 09 23:21:20.568415 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:21:20.568396 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:21:21.121986 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:21.121943 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:21:21.122385 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:21.122018 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:21:21.122385 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:21.122035 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:21:21.122385 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:21.122052 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:21:21.122490 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:21.122392 2561 scope.go:117] "RemoveContainer" containerID="a2980bece3135c5fa297152bb0a2f81aa29d538e94e79026a9d33e90db2103d8" Sep 09 23:21:21.122579 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:21:21.122561 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:21:26.011245 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:26.011188 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:21:26.011684 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:26.011275 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:21:26.011684 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:26.011292 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:21:26.011684 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:26.011304 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:21:26.011684 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:26.011673 2561 scope.go:117] "RemoveContainer" containerID="a30728a8c607be25a1fe9027ab4ae1d4c22addd6a6bafc8f1db68139296cdbd0" Sep 09 23:21:26.011900 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:21:26.011842 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:21:32.194975 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:32.194938 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j"] Sep 09 23:21:32.201319 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:32.201288 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:21:32.203835 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:32.203812 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 09 23:21:32.208548 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:32.208517 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j"] Sep 09 23:21:32.317305 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:32.317267 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j\" (UID: \"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:21:32.317305 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:32.317305 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j\" (UID: \"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:21:32.317560 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:32.317437 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9-dshm\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j\" (UID: \"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:21:32.317560 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:32.317456 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j\" (UID: \"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:21:32.317560 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:32.317475 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9-home\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j\" (UID: \"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:21:32.317560 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:32.317496 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2gtf\" (UniqueName: \"kubernetes.io/projected/2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9-kube-api-access-x2gtf\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j\" (UID: \"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:21:32.417991 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:32.417944 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9-dshm\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j\" (UID: \"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:21:32.417991 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:32.417991 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j\" (UID: \"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:21:32.418254 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:32.418121 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9-home\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j\" (UID: \"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:21:32.418254 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:32.418167 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2gtf\" (UniqueName: \"kubernetes.io/projected/2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9-kube-api-access-x2gtf\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j\" (UID: \"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:21:32.418254 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:32.418195 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j\" (UID: \"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:21:32.418254 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:32.418217 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j\" (UID: \"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:21:32.418447 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:32.418396 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j\" (UID: \"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:21:32.418507 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:32.418471 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9-home\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j\" (UID: \"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:21:32.418572 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:32.418550 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j\" (UID: \"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:21:32.420584 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:32.420553 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9-dshm\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j\" (UID: \"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:21:32.420816 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:32.420798 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j\" (UID: \"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:21:32.425238 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:32.425219 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2gtf\" (UniqueName: \"kubernetes.io/projected/2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9-kube-api-access-x2gtf\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j\" (UID: \"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:21:32.514592 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:32.514496 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:21:32.787456 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:32.787323 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j"] Sep 09 23:21:32.789737 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:21:32.789704 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ddcfdf7_7daf_4b30_a3cf_18d5fe84c4b9.slice/crio-1b3838a9250c73e687f0937e61c0836681e98a170657a9d2123de10b9004d59b WatchSource:0}: Error finding container 1b3838a9250c73e687f0937e61c0836681e98a170657a9d2123de10b9004d59b: Status 404 returned error can't find the container with id 1b3838a9250c73e687f0937e61c0836681e98a170657a9d2123de10b9004d59b Sep 09 23:21:33.632590 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:33.632512 2561 generic.go:356] "Generic (PLEG): container finished" podID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" containerID="e25449c29786da7242fe2b370d31ebc5c47d8a1f53957b0b8eb894473ddd464c" exitCode=0 Sep 09 23:21:33.632965 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:33.632601 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" event={"ID":"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9","Type":"ContainerDied","Data":"e25449c29786da7242fe2b370d31ebc5c47d8a1f53957b0b8eb894473ddd464c"} Sep 09 23:21:33.632965 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:33.632668 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" event={"ID":"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9","Type":"ContainerStarted","Data":"1b3838a9250c73e687f0937e61c0836681e98a170657a9d2123de10b9004d59b"} Sep 09 23:21:33.632965 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:33.632790 2561 scope.go:117] "RemoveContainer" containerID="e25449c29786da7242fe2b370d31ebc5c47d8a1f53957b0b8eb894473ddd464c" Sep 09 23:21:34.638958 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:34.638920 2561 generic.go:356] "Generic (PLEG): container finished" podID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" containerID="08489c6cb29cd98a4d1dd226a1e152be3bc940c511005d72c66ccb170ff21e80" exitCode=0 Sep 09 23:21:34.639393 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:34.639011 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" event={"ID":"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9","Type":"ContainerDied","Data":"08489c6cb29cd98a4d1dd226a1e152be3bc940c511005d72c66ccb170ff21e80"} Sep 09 23:21:34.639393 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:34.639077 2561 scope.go:117] "RemoveContainer" containerID="e25449c29786da7242fe2b370d31ebc5c47d8a1f53957b0b8eb894473ddd464c" Sep 09 23:21:34.639507 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:34.639398 2561 scope.go:117] "RemoveContainer" containerID="08489c6cb29cd98a4d1dd226a1e152be3bc940c511005d72c66ccb170ff21e80" Sep 09 23:21:34.639606 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:21:34.639587 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:21:34.804658 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:34.804619 2561 scope.go:117] "RemoveContainer" containerID="a2980bece3135c5fa297152bb0a2f81aa29d538e94e79026a9d33e90db2103d8" Sep 09 23:21:34.804846 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:21:34.804824 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:21:35.645330 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:35.645300 2561 scope.go:117] "RemoveContainer" containerID="08489c6cb29cd98a4d1dd226a1e152be3bc940c511005d72c66ccb170ff21e80" Sep 09 23:21:35.645722 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:21:35.645516 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:21:37.804512 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:37.804480 2561 scope.go:117] "RemoveContainer" containerID="a30728a8c607be25a1fe9027ab4ae1d4c22addd6a6bafc8f1db68139296cdbd0" Sep 09 23:21:37.804917 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:21:37.804731 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:21:42.514836 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:42.514738 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:21:42.514836 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:42.514820 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:21:42.515292 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:42.514842 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:21:42.515292 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:42.514891 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:21:42.515292 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:42.515280 2561 scope.go:117] "RemoveContainer" containerID="08489c6cb29cd98a4d1dd226a1e152be3bc940c511005d72c66ccb170ff21e80" Sep 09 23:21:42.515511 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:21:42.515490 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:21:42.679126 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:42.679095 2561 scope.go:117] "RemoveContainer" containerID="08489c6cb29cd98a4d1dd226a1e152be3bc940c511005d72c66ccb170ff21e80" Sep 09 23:21:42.679313 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:21:42.679295 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:21:45.814977 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:45.814949 2561 scope.go:117] "RemoveContainer" containerID="a2980bece3135c5fa297152bb0a2f81aa29d538e94e79026a9d33e90db2103d8" Sep 09 23:21:45.815471 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:21:45.815177 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:21:51.767305 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:51.767278 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2fs7f_0a127828-4a97-40cf-b8ae-f5ecd20f97c6/console-operator/1.log" Sep 09 23:21:51.767963 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:51.767779 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2fs7f_0a127828-4a97-40cf-b8ae-f5ecd20f97c6/console-operator/1.log" Sep 09 23:21:51.807232 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:51.807204 2561 scope.go:117] "RemoveContainer" containerID="a30728a8c607be25a1fe9027ab4ae1d4c22addd6a6bafc8f1db68139296cdbd0" Sep 09 23:21:51.807491 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:21:51.807447 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:21:52.804120 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:52.804089 2561 scope.go:117] "RemoveContainer" containerID="08489c6cb29cd98a4d1dd226a1e152be3bc940c511005d72c66ccb170ff21e80" Sep 09 23:21:53.731783 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:53.731747 2561 generic.go:356] "Generic (PLEG): container finished" podID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" containerID="06a261fd580aac01ba2d75f89fd54400f24966c8c787e0f775c021e3fff3aced" exitCode=0 Sep 09 23:21:53.731981 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:53.731789 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" event={"ID":"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9","Type":"ContainerDied","Data":"06a261fd580aac01ba2d75f89fd54400f24966c8c787e0f775c021e3fff3aced"} Sep 09 23:21:53.731981 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:53.731876 2561 scope.go:117] "RemoveContainer" containerID="08489c6cb29cd98a4d1dd226a1e152be3bc940c511005d72c66ccb170ff21e80" Sep 09 23:21:53.732275 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:21:53.732252 2561 scope.go:117] "RemoveContainer" containerID="06a261fd580aac01ba2d75f89fd54400f24966c8c787e0f775c021e3fff3aced" Sep 09 23:21:53.732463 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:21:53.732445 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:22:00.805344 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:00.805256 2561 scope.go:117] "RemoveContainer" containerID="a2980bece3135c5fa297152bb0a2f81aa29d538e94e79026a9d33e90db2103d8" Sep 09 23:22:01.777137 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:01.777105 2561 generic.go:356] "Generic (PLEG): container finished" podID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" containerID="8e0f40fbbfe1a3de0d0e59ffa33a767de4d4e6eaa2edc0a486b7f425ddcb3d29" exitCode=0 Sep 09 23:22:01.777335 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:01.777179 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" event={"ID":"ecff2538-a36b-41ca-8484-c5f54bb2cc89","Type":"ContainerDied","Data":"8e0f40fbbfe1a3de0d0e59ffa33a767de4d4e6eaa2edc0a486b7f425ddcb3d29"} Sep 09 23:22:01.777335 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:01.777241 2561 scope.go:117] "RemoveContainer" containerID="a2980bece3135c5fa297152bb0a2f81aa29d538e94e79026a9d33e90db2103d8" Sep 09 23:22:01.777710 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:01.777694 2561 scope.go:117] "RemoveContainer" containerID="8e0f40fbbfe1a3de0d0e59ffa33a767de4d4e6eaa2edc0a486b7f425ddcb3d29" Sep 09 23:22:01.777943 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:22:01.777924 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:22:02.514870 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:02.514801 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:22:02.515279 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:02.514939 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:22:02.515279 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:02.514959 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:22:02.515279 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:02.514974 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:22:02.515386 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:02.515301 2561 scope.go:117] "RemoveContainer" containerID="06a261fd580aac01ba2d75f89fd54400f24966c8c787e0f775c021e3fff3aced" Sep 09 23:22:02.515520 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:22:02.515491 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:22:06.804331 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:06.804303 2561 scope.go:117] "RemoveContainer" containerID="a30728a8c607be25a1fe9027ab4ae1d4c22addd6a6bafc8f1db68139296cdbd0" Sep 09 23:22:07.810198 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:07.810167 2561 generic.go:356] "Generic (PLEG): container finished" podID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" containerID="f20ad70732adefd9d3a7bf569ef74d534ed14233009913e50270dee6f8cb20f1" exitCode=0 Sep 09 23:22:07.810198 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:07.810200 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" event={"ID":"bbc27533-85d7-4f55-8f45-dcd10685b8f6","Type":"ContainerDied","Data":"f20ad70732adefd9d3a7bf569ef74d534ed14233009913e50270dee6f8cb20f1"} Sep 09 23:22:07.810707 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:07.810240 2561 scope.go:117] "RemoveContainer" containerID="a30728a8c607be25a1fe9027ab4ae1d4c22addd6a6bafc8f1db68139296cdbd0" Sep 09 23:22:07.810707 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:07.810556 2561 scope.go:117] "RemoveContainer" containerID="f20ad70732adefd9d3a7bf569ef74d534ed14233009913e50270dee6f8cb20f1" Sep 09 23:22:07.810790 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:22:07.810762 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:22:11.121876 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:11.121818 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:22:11.122294 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:11.121917 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:22:11.122294 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:11.121938 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:22:11.122294 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:11.121951 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:22:11.122424 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:11.122304 2561 scope.go:117] "RemoveContainer" containerID="8e0f40fbbfe1a3de0d0e59ffa33a767de4d4e6eaa2edc0a486b7f425ddcb3d29" Sep 09 23:22:11.122505 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:22:11.122486 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:22:16.010485 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:16.010436 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:22:16.010926 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:16.010513 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:22:16.010926 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:16.010533 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:22:16.010926 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:16.010546 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:22:16.011074 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:16.010947 2561 scope.go:117] "RemoveContainer" containerID="f20ad70732adefd9d3a7bf569ef74d534ed14233009913e50270dee6f8cb20f1" Sep 09 23:22:16.011167 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:22:16.011148 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:22:16.804558 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:16.804519 2561 scope.go:117] "RemoveContainer" containerID="06a261fd580aac01ba2d75f89fd54400f24966c8c787e0f775c021e3fff3aced" Sep 09 23:22:17.866321 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:17.866286 2561 generic.go:356] "Generic (PLEG): container finished" podID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" containerID="6c6dfd03bbd258b3a78f9916f576de45f1926bacdab71a794016c22227eee24d" exitCode=0 Sep 09 23:22:17.866743 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:17.866331 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" event={"ID":"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9","Type":"ContainerDied","Data":"6c6dfd03bbd258b3a78f9916f576de45f1926bacdab71a794016c22227eee24d"} Sep 09 23:22:17.866743 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:17.866398 2561 scope.go:117] "RemoveContainer" containerID="06a261fd580aac01ba2d75f89fd54400f24966c8c787e0f775c021e3fff3aced" Sep 09 23:22:17.866875 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:17.866764 2561 scope.go:117] "RemoveContainer" containerID="6c6dfd03bbd258b3a78f9916f576de45f1926bacdab71a794016c22227eee24d" Sep 09 23:22:17.867006 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:22:17.866983 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:22:22.514582 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:22.514542 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:22:22.515034 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:22.514619 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:22:22.515034 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:22.514642 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:22:22.515034 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:22.514660 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:22:22.515157 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:22.515118 2561 scope.go:117] "RemoveContainer" containerID="6c6dfd03bbd258b3a78f9916f576de45f1926bacdab71a794016c22227eee24d" Sep 09 23:22:22.515317 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:22:22.515297 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:22:24.803959 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:24.803921 2561 scope.go:117] "RemoveContainer" containerID="8e0f40fbbfe1a3de0d0e59ffa33a767de4d4e6eaa2edc0a486b7f425ddcb3d29" Sep 09 23:22:24.804433 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:22:24.804167 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:22:27.804847 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:27.804803 2561 scope.go:117] "RemoveContainer" containerID="f20ad70732adefd9d3a7bf569ef74d534ed14233009913e50270dee6f8cb20f1" Sep 09 23:22:27.805389 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:22:27.805078 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:22:35.804443 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:35.804414 2561 scope.go:117] "RemoveContainer" containerID="6c6dfd03bbd258b3a78f9916f576de45f1926bacdab71a794016c22227eee24d" Sep 09 23:22:35.804882 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:22:35.804606 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:22:39.804515 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:39.804484 2561 scope.go:117] "RemoveContainer" containerID="8e0f40fbbfe1a3de0d0e59ffa33a767de4d4e6eaa2edc0a486b7f425ddcb3d29" Sep 09 23:22:39.804942 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:22:39.804675 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:22:42.803989 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:42.803948 2561 scope.go:117] "RemoveContainer" containerID="f20ad70732adefd9d3a7bf569ef74d534ed14233009913e50270dee6f8cb20f1" Sep 09 23:22:42.804471 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:22:42.804117 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:22:46.804383 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:46.804343 2561 scope.go:117] "RemoveContainer" containerID="6c6dfd03bbd258b3a78f9916f576de45f1926bacdab71a794016c22227eee24d" Sep 09 23:22:46.804780 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:22:46.804539 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:22:54.804343 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:54.804306 2561 scope.go:117] "RemoveContainer" containerID="8e0f40fbbfe1a3de0d0e59ffa33a767de4d4e6eaa2edc0a486b7f425ddcb3d29" Sep 09 23:22:54.804751 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:22:54.804495 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:22:57.804794 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:57.804757 2561 scope.go:117] "RemoveContainer" containerID="f20ad70732adefd9d3a7bf569ef74d534ed14233009913e50270dee6f8cb20f1" Sep 09 23:22:57.805245 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:22:57.805007 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:22:59.804762 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:22:59.804732 2561 scope.go:117] "RemoveContainer" containerID="6c6dfd03bbd258b3a78f9916f576de45f1926bacdab71a794016c22227eee24d" Sep 09 23:23:00.078226 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:00.078194 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" event={"ID":"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9","Type":"ContainerStarted","Data":"046790c1ba888dc8097bcc68591fc9f1b5265513485e5280a7d77e1591667209"} Sep 09 23:23:00.093146 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:00.093102 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podStartSLOduration=87.165674323 podStartE2EDuration="1m28.093087479s" totalImagesPullingTime="927.413156ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:21:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:23:00.092074669 +0000 UTC m=+968.777858091" watchObservedRunningTime="2026-09-09 23:23:00.093087479 +0000 UTC m=+968.778870880" Sep 09 23:23:01.084920 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:01.084879 2561 generic.go:356] "Generic (PLEG): container finished" podID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" containerID="046790c1ba888dc8097bcc68591fc9f1b5265513485e5280a7d77e1591667209" exitCode=0 Sep 09 23:23:01.085336 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:01.084929 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" event={"ID":"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9","Type":"ContainerDied","Data":"046790c1ba888dc8097bcc68591fc9f1b5265513485e5280a7d77e1591667209"} Sep 09 23:23:01.085336 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:01.085039 2561 scope.go:117] "RemoveContainer" containerID="6c6dfd03bbd258b3a78f9916f576de45f1926bacdab71a794016c22227eee24d" Sep 09 23:23:01.085454 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:01.085404 2561 scope.go:117] "RemoveContainer" containerID="046790c1ba888dc8097bcc68591fc9f1b5265513485e5280a7d77e1591667209" Sep 09 23:23:01.085629 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:23:01.085605 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:23:02.515523 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:02.515485 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:23:02.515960 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:02.515554 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:23:02.515960 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:02.515569 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:23:02.515960 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:02.515582 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:23:02.516072 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:02.515980 2561 scope.go:117] "RemoveContainer" containerID="046790c1ba888dc8097bcc68591fc9f1b5265513485e5280a7d77e1591667209" Sep 09 23:23:02.516177 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:23:02.516158 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:23:06.804553 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:06.804521 2561 scope.go:117] "RemoveContainer" containerID="8e0f40fbbfe1a3de0d0e59ffa33a767de4d4e6eaa2edc0a486b7f425ddcb3d29" Sep 09 23:23:06.805122 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:23:06.804704 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:23:11.806965 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:11.806931 2561 scope.go:117] "RemoveContainer" containerID="f20ad70732adefd9d3a7bf569ef74d534ed14233009913e50270dee6f8cb20f1" Sep 09 23:23:11.807379 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:23:11.807095 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:23:15.804154 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:15.804077 2561 scope.go:117] "RemoveContainer" containerID="046790c1ba888dc8097bcc68591fc9f1b5265513485e5280a7d77e1591667209" Sep 09 23:23:15.804591 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:23:15.804257 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:23:20.804777 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:20.804748 2561 scope.go:117] "RemoveContainer" containerID="8e0f40fbbfe1a3de0d0e59ffa33a767de4d4e6eaa2edc0a486b7f425ddcb3d29" Sep 09 23:23:20.805168 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:23:20.804942 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:23:23.804619 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:23.804587 2561 scope.go:117] "RemoveContainer" containerID="f20ad70732adefd9d3a7bf569ef74d534ed14233009913e50270dee6f8cb20f1" Sep 09 23:23:23.805029 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:23:23.804783 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:23:26.803769 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:26.803730 2561 scope.go:117] "RemoveContainer" containerID="046790c1ba888dc8097bcc68591fc9f1b5265513485e5280a7d77e1591667209" Sep 09 23:23:26.804377 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:23:26.804016 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:23:35.804221 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:35.804186 2561 scope.go:117] "RemoveContainer" containerID="f20ad70732adefd9d3a7bf569ef74d534ed14233009913e50270dee6f8cb20f1" Sep 09 23:23:35.804758 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:35.804342 2561 scope.go:117] "RemoveContainer" containerID="8e0f40fbbfe1a3de0d0e59ffa33a767de4d4e6eaa2edc0a486b7f425ddcb3d29" Sep 09 23:23:36.264061 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:36.264030 2561 generic.go:356] "Generic (PLEG): container finished" podID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" containerID="8f46ce3935c3b9dee894d62de478a13ebfa89401c11f98949b1061cde719cb9c" exitCode=0 Sep 09 23:23:36.264263 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:36.264106 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" event={"ID":"ecff2538-a36b-41ca-8484-c5f54bb2cc89","Type":"ContainerDied","Data":"8f46ce3935c3b9dee894d62de478a13ebfa89401c11f98949b1061cde719cb9c"} Sep 09 23:23:36.264263 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:36.264171 2561 scope.go:117] "RemoveContainer" containerID="8e0f40fbbfe1a3de0d0e59ffa33a767de4d4e6eaa2edc0a486b7f425ddcb3d29" Sep 09 23:23:36.264580 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:36.264563 2561 scope.go:117] "RemoveContainer" containerID="8f46ce3935c3b9dee894d62de478a13ebfa89401c11f98949b1061cde719cb9c" Sep 09 23:23:36.264839 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:23:36.264801 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:23:36.266290 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:36.266262 2561 generic.go:356] "Generic (PLEG): container finished" podID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" containerID="bca67ebbb21c9195e26494183dbf2b704229cf1e45c09e47ab8a3a95506cb6eb" exitCode=0 Sep 09 23:23:36.266370 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:36.266331 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" event={"ID":"bbc27533-85d7-4f55-8f45-dcd10685b8f6","Type":"ContainerDied","Data":"bca67ebbb21c9195e26494183dbf2b704229cf1e45c09e47ab8a3a95506cb6eb"} Sep 09 23:23:36.266665 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:36.266652 2561 scope.go:117] "RemoveContainer" containerID="bca67ebbb21c9195e26494183dbf2b704229cf1e45c09e47ab8a3a95506cb6eb" Sep 09 23:23:36.266826 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:23:36.266811 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:23:36.275343 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:36.275319 2561 scope.go:117] "RemoveContainer" containerID="f20ad70732adefd9d3a7bf569ef74d534ed14233009913e50270dee6f8cb20f1" Sep 09 23:23:39.805847 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:39.804914 2561 scope.go:117] "RemoveContainer" containerID="046790c1ba888dc8097bcc68591fc9f1b5265513485e5280a7d77e1591667209" Sep 09 23:23:39.805847 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:23:39.805378 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:23:41.122141 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:41.122097 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:23:41.122562 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:41.122180 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:23:41.122562 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:41.122203 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:23:41.122562 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:41.122223 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" Sep 09 23:23:41.122689 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:41.122604 2561 scope.go:117] "RemoveContainer" containerID="8f46ce3935c3b9dee894d62de478a13ebfa89401c11f98949b1061cde719cb9c" Sep 09 23:23:41.122789 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:23:41.122770 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:23:46.010619 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:46.010568 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:23:46.011055 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:46.010649 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:23:46.011055 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:46.010664 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:23:46.011055 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:46.010677 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" Sep 09 23:23:46.011055 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:46.011049 2561 scope.go:117] "RemoveContainer" containerID="bca67ebbb21c9195e26494183dbf2b704229cf1e45c09e47ab8a3a95506cb6eb" Sep 09 23:23:46.011233 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:23:46.011215 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:23:51.808018 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:51.807985 2561 scope.go:117] "RemoveContainer" containerID="046790c1ba888dc8097bcc68591fc9f1b5265513485e5280a7d77e1591667209" Sep 09 23:23:51.808524 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:23:51.808231 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:23:55.810405 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:55.810376 2561 scope.go:117] "RemoveContainer" containerID="8f46ce3935c3b9dee894d62de478a13ebfa89401c11f98949b1061cde719cb9c" Sep 09 23:23:55.810801 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:23:55.810570 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:23:57.804407 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:23:57.804365 2561 scope.go:117] "RemoveContainer" containerID="bca67ebbb21c9195e26494183dbf2b704229cf1e45c09e47ab8a3a95506cb6eb" Sep 09 23:23:57.804936 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:23:57.804566 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:24:04.804122 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:24:04.804087 2561 scope.go:117] "RemoveContainer" containerID="046790c1ba888dc8097bcc68591fc9f1b5265513485e5280a7d77e1591667209" Sep 09 23:24:04.804628 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:24:04.804278 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:24:06.804750 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:24:06.804717 2561 scope.go:117] "RemoveContainer" containerID="8f46ce3935c3b9dee894d62de478a13ebfa89401c11f98949b1061cde719cb9c" Sep 09 23:24:06.805174 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:24:06.804907 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:24:08.804641 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:24:08.804608 2561 scope.go:117] "RemoveContainer" containerID="bca67ebbb21c9195e26494183dbf2b704229cf1e45c09e47ab8a3a95506cb6eb" Sep 09 23:24:08.805070 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:24:08.804781 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:24:17.804675 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:24:17.804634 2561 scope.go:117] "RemoveContainer" containerID="8f46ce3935c3b9dee894d62de478a13ebfa89401c11f98949b1061cde719cb9c" Sep 09 23:24:17.805227 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:24:17.804989 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:24:19.804578 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:24:19.804551 2561 scope.go:117] "RemoveContainer" containerID="046790c1ba888dc8097bcc68591fc9f1b5265513485e5280a7d77e1591667209" Sep 09 23:24:19.805033 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:24:19.804706 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:24:21.813249 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:24:21.809898 2561 scope.go:117] "RemoveContainer" containerID="bca67ebbb21c9195e26494183dbf2b704229cf1e45c09e47ab8a3a95506cb6eb" Sep 09 23:24:21.813249 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:24:21.810271 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:24:30.804507 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:24:30.804469 2561 scope.go:117] "RemoveContainer" containerID="8f46ce3935c3b9dee894d62de478a13ebfa89401c11f98949b1061cde719cb9c" Sep 09 23:24:30.804977 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:24:30.804710 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:24:32.803999 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:24:32.803968 2561 scope.go:117] "RemoveContainer" containerID="046790c1ba888dc8097bcc68591fc9f1b5265513485e5280a7d77e1591667209" Sep 09 23:24:33.551612 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:24:33.551579 2561 generic.go:356] "Generic (PLEG): container finished" podID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" containerID="575ffaae2f5075dde2eb85100cf8f6a1b1a00a177163bfeb735cb37448293405" exitCode=0 Sep 09 23:24:33.551778 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:24:33.551640 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" event={"ID":"2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9","Type":"ContainerDied","Data":"575ffaae2f5075dde2eb85100cf8f6a1b1a00a177163bfeb735cb37448293405"} Sep 09 23:24:33.551778 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:24:33.551696 2561 scope.go:117] "RemoveContainer" containerID="046790c1ba888dc8097bcc68591fc9f1b5265513485e5280a7d77e1591667209" Sep 09 23:24:33.552128 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:24:33.552112 2561 scope.go:117] "RemoveContainer" containerID="575ffaae2f5075dde2eb85100cf8f6a1b1a00a177163bfeb735cb37448293405" Sep 09 23:24:33.552344 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:24:33.552319 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:24:33.804810 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:24:33.804715 2561 scope.go:117] "RemoveContainer" containerID="bca67ebbb21c9195e26494183dbf2b704229cf1e45c09e47ab8a3a95506cb6eb" Sep 09 23:24:33.805254 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:24:33.804972 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:24:42.515546 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:24:42.515463 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:24:42.515989 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:24:42.515560 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:24:42.515989 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:24:42.515609 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:24:42.515989 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:24:42.515625 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" Sep 09 23:24:42.515989 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:24:42.515897 2561 scope.go:117] "RemoveContainer" containerID="575ffaae2f5075dde2eb85100cf8f6a1b1a00a177163bfeb735cb37448293405" Sep 09 23:24:42.516128 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:24:42.516061 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:24:42.596366 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:24:42.596332 2561 scope.go:117] "RemoveContainer" containerID="575ffaae2f5075dde2eb85100cf8f6a1b1a00a177163bfeb735cb37448293405" Sep 09 23:24:42.596600 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:24:42.596578 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:24:43.804949 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:24:43.804907 2561 scope.go:117] "RemoveContainer" containerID="8f46ce3935c3b9dee894d62de478a13ebfa89401c11f98949b1061cde719cb9c" Sep 09 23:24:43.805444 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:24:43.805191 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:24:47.804699 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:24:47.804670 2561 scope.go:117] "RemoveContainer" containerID="bca67ebbb21c9195e26494183dbf2b704229cf1e45c09e47ab8a3a95506cb6eb" Sep 09 23:24:47.805186 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:24:47.804824 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:24:53.803976 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:24:53.803944 2561 scope.go:117] "RemoveContainer" containerID="575ffaae2f5075dde2eb85100cf8f6a1b1a00a177163bfeb735cb37448293405" Sep 09 23:24:53.804470 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:24:53.804109 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:24:57.804673 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:24:57.804639 2561 scope.go:117] "RemoveContainer" containerID="8f46ce3935c3b9dee894d62de478a13ebfa89401c11f98949b1061cde719cb9c" Sep 09 23:24:57.805212 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:24:57.804829 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:25:01.806666 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:01.806639 2561 scope.go:117] "RemoveContainer" containerID="bca67ebbb21c9195e26494183dbf2b704229cf1e45c09e47ab8a3a95506cb6eb" Sep 09 23:25:01.807073 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:25:01.806825 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:25:05.804242 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:05.804210 2561 scope.go:117] "RemoveContainer" containerID="575ffaae2f5075dde2eb85100cf8f6a1b1a00a177163bfeb735cb37448293405" Sep 09 23:25:05.804606 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:25:05.804391 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:25:10.804671 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:10.804642 2561 scope.go:117] "RemoveContainer" containerID="8f46ce3935c3b9dee894d62de478a13ebfa89401c11f98949b1061cde719cb9c" Sep 09 23:25:10.805084 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:25:10.804819 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:25:16.804104 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:16.804068 2561 scope.go:117] "RemoveContainer" containerID="bca67ebbb21c9195e26494183dbf2b704229cf1e45c09e47ab8a3a95506cb6eb" Sep 09 23:25:16.804569 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:25:16.804285 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:25:18.803902 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:18.803840 2561 scope.go:117] "RemoveContainer" containerID="575ffaae2f5075dde2eb85100cf8f6a1b1a00a177163bfeb735cb37448293405" Sep 09 23:25:18.804395 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:25:18.804043 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:25:21.807415 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:21.807376 2561 scope.go:117] "RemoveContainer" containerID="8f46ce3935c3b9dee894d62de478a13ebfa89401c11f98949b1061cde719cb9c" Sep 09 23:25:21.807795 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:25:21.807636 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:25:27.318169 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:27.318141 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9/main/5.log" Sep 09 23:25:27.429537 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:27.429504 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9/main/5.log" Sep 09 23:25:29.804838 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:29.804801 2561 scope.go:117] "RemoveContainer" containerID="bca67ebbb21c9195e26494183dbf2b704229cf1e45c09e47ab8a3a95506cb6eb" Sep 09 23:25:29.805320 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:25:29.805062 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:25:31.806304 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:31.806271 2561 scope.go:117] "RemoveContainer" containerID="575ffaae2f5075dde2eb85100cf8f6a1b1a00a177163bfeb735cb37448293405" Sep 09 23:25:31.806754 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:25:31.806437 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:25:33.804371 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:33.804342 2561 scope.go:117] "RemoveContainer" containerID="8f46ce3935c3b9dee894d62de478a13ebfa89401c11f98949b1061cde719cb9c" Sep 09 23:25:33.804897 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:25:33.804503 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:25:33.836680 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:33.836652 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-558d5b9f86-xpn66_e8e5f1bf-f57f-4965-bc29-43e217345fcd/manager/0.log" Sep 09 23:25:34.179726 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:34.179702 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c5b764bdf-dxhww_b3581d49-a5b5-4b09-96aa-9f58bcc64c9b/copy-manifests/0.log" Sep 09 23:25:34.185827 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:34.185797 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c5b764bdf-dxhww_b3581d49-a5b5-4b09-96aa-9f58bcc64c9b/manager/0.log" Sep 09 23:25:34.761442 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:34.761404 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-f576956d9-mzv7x_8dc3549a-cfbd-4890-b4c2-78394425a147/maas-api/0.log" Sep 09 23:25:34.867344 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:34.867318 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-t9k6z_4a837c5d-99aa-40fd-8959-d51cf2da2122/postgres/0.log" Sep 09 23:25:36.785165 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:36.785128 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-jnwpf_5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae/discovery/0.log" Sep 09 23:25:37.234952 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:37.234928 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-h555r_eb8317f8-8b9c-4309-94da-ae5b484e14ab/payload-pre-processing/0.log" Sep 09 23:25:37.457088 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:37.457064 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b46465886-mcs6c_6fa8702e-a63e-43aa-907c-e411e4af1b11/router/0.log" Sep 09 23:25:37.786708 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:37.786677 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-859856b874-678lf_ecff2538-a36b-41ca-8484-c5f54bb2cc89/main/5.log" Sep 09 23:25:37.896872 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:37.896828 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-6685d86975-8tlsl_bbc27533-85d7-4f55-8f45-dcd10685b8f6/main/5.log" Sep 09 23:25:38.336408 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:38.336370 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9/main/5.log" Sep 09 23:25:43.804412 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:43.804378 2561 scope.go:117] "RemoveContainer" containerID="bca67ebbb21c9195e26494183dbf2b704229cf1e45c09e47ab8a3a95506cb6eb" Sep 09 23:25:43.804792 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:25:43.804605 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:25:44.803694 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:44.803667 2561 scope.go:117] "RemoveContainer" containerID="575ffaae2f5075dde2eb85100cf8f6a1b1a00a177163bfeb735cb37448293405" Sep 09 23:25:44.803884 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:25:44.803839 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:25:46.670665 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:46.670632 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gw2dp_31b8a991-a7b8-4cae-9125-69fcd001f700/global-pull-secret-syncer/0.log" Sep 09 23:25:46.769418 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:46.769390 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-z9xqn_d5402f8c-eefb-4fb3-9742-b96b8207198f/konnectivity-agent/0.log" Sep 09 23:25:46.919229 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:46.919204 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-192.ec2.internal_ef36ec20e9fe178e1d7d8c3ae65f1ea2/haproxy/0.log" Sep 09 23:25:48.803767 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:48.803733 2561 scope.go:117] "RemoveContainer" containerID="8f46ce3935c3b9dee894d62de478a13ebfa89401c11f98949b1061cde719cb9c" Sep 09 23:25:48.804256 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:25:48.803974 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:25:52.996511 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:52.996480 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-fjn7n_6929f83e-bb61-4ee4-b7e2-54785f9746cf/kube-state-metrics/0.log" Sep 09 23:25:53.016600 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:53.016577 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-fjn7n_6929f83e-bb61-4ee4-b7e2-54785f9746cf/kube-rbac-proxy-main/0.log" Sep 09 23:25:53.036316 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:53.036292 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-fjn7n_6929f83e-bb61-4ee4-b7e2-54785f9746cf/kube-rbac-proxy-self/0.log" Sep 09 23:25:53.124011 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:53.123941 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-78brr_eeb3e51b-bd93-4733-8149-977b136d46b0/node-exporter/0.log" Sep 09 23:25:53.142352 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:53.142330 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-78brr_eeb3e51b-bd93-4733-8149-977b136d46b0/kube-rbac-proxy/0.log" Sep 09 23:25:53.160586 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:53.160567 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-78brr_eeb3e51b-bd93-4733-8149-977b136d46b0/init-textfile/0.log" Sep 09 23:25:53.336359 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:53.336334 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-v6wvd_d33d9ae1-397a-4288-beea-c41cb515a274/kube-rbac-proxy-main/0.log" Sep 09 23:25:53.355648 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:53.355622 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-v6wvd_d33d9ae1-397a-4288-beea-c41cb515a274/kube-rbac-proxy-self/0.log" Sep 09 23:25:53.376449 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:53.376397 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-v6wvd_d33d9ae1-397a-4288-beea-c41cb515a274/openshift-state-metrics/0.log" Sep 09 23:25:53.416210 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:53.416189 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cda93acf-5e54-430b-8187-b2d6265ff0d0/prometheus/0.log" Sep 09 23:25:53.438468 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:53.438450 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cda93acf-5e54-430b-8187-b2d6265ff0d0/config-reloader/0.log" Sep 09 23:25:53.458735 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:53.458716 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cda93acf-5e54-430b-8187-b2d6265ff0d0/thanos-sidecar/0.log" Sep 09 23:25:53.478774 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:53.478755 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cda93acf-5e54-430b-8187-b2d6265ff0d0/kube-rbac-proxy-web/0.log" Sep 09 23:25:53.502306 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:53.502287 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cda93acf-5e54-430b-8187-b2d6265ff0d0/kube-rbac-proxy/0.log" Sep 09 23:25:53.523300 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:53.523280 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cda93acf-5e54-430b-8187-b2d6265ff0d0/kube-rbac-proxy-thanos/0.log" Sep 09 23:25:53.542575 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:53.542557 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cda93acf-5e54-430b-8187-b2d6265ff0d0/init-config-reloader/0.log" Sep 09 23:25:53.738186 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:53.738160 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bff948996-pb87r_eb5a4583-4f79-4391-80c1-72c88355e67b/thanos-query/0.log" Sep 09 23:25:53.755887 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:53.755844 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bff948996-pb87r_eb5a4583-4f79-4391-80c1-72c88355e67b/kube-rbac-proxy-web/0.log" Sep 09 23:25:53.777547 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:53.777529 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bff948996-pb87r_eb5a4583-4f79-4391-80c1-72c88355e67b/kube-rbac-proxy/0.log" Sep 09 23:25:53.796211 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:53.796188 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bff948996-pb87r_eb5a4583-4f79-4391-80c1-72c88355e67b/prom-label-proxy/0.log" Sep 09 23:25:53.815288 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:53.815272 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bff948996-pb87r_eb5a4583-4f79-4391-80c1-72c88355e67b/kube-rbac-proxy-rules/0.log" Sep 09 23:25:53.835015 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:53.834993 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bff948996-pb87r_eb5a4583-4f79-4391-80c1-72c88355e67b/kube-rbac-proxy-metrics/0.log" Sep 09 23:25:56.037659 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:56.037626 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2fs7f_0a127828-4a97-40cf-b8ae-f5ecd20f97c6/console-operator/1.log" Sep 09 23:25:56.042765 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:56.042742 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2fs7f_0a127828-4a97-40cf-b8ae-f5ecd20f97c6/console-operator/2.log" Sep 09 23:25:56.584743 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:56.584719 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-74dcdb946c-2czls_6ed8e9b7-ad65-44eb-a256-f23db3a47011/console/0.log" Sep 09 23:25:57.064510 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.064482 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-njqsq/perf-node-gather-daemonset-629q6"] Sep 09 23:25:57.068306 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.068283 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-njqsq/perf-node-gather-daemonset-629q6" Sep 09 23:25:57.070665 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.070646 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-njqsq\"/\"openshift-service-ca.crt\"" Sep 09 23:25:57.070771 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.070752 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-njqsq\"/\"kube-root-ca.crt\"" Sep 09 23:25:57.071560 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.071546 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-njqsq\"/\"default-dockercfg-rjsgt\"" Sep 09 23:25:57.077338 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.077318 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-njqsq/perf-node-gather-daemonset-629q6"] Sep 09 23:25:57.141105 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.141077 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d54c2ead-5bf7-48d4-baa8-a00db0adfb86-proc\") pod \"perf-node-gather-daemonset-629q6\" (UID: \"d54c2ead-5bf7-48d4-baa8-a00db0adfb86\") " pod="openshift-must-gather-njqsq/perf-node-gather-daemonset-629q6" Sep 09 23:25:57.141105 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.141106 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d54c2ead-5bf7-48d4-baa8-a00db0adfb86-podres\") pod \"perf-node-gather-daemonset-629q6\" (UID: \"d54c2ead-5bf7-48d4-baa8-a00db0adfb86\") " pod="openshift-must-gather-njqsq/perf-node-gather-daemonset-629q6" Sep 09 23:25:57.141257 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.141130 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr9s6\" (UniqueName: \"kubernetes.io/projected/d54c2ead-5bf7-48d4-baa8-a00db0adfb86-kube-api-access-jr9s6\") pod \"perf-node-gather-daemonset-629q6\" (UID: \"d54c2ead-5bf7-48d4-baa8-a00db0adfb86\") " pod="openshift-must-gather-njqsq/perf-node-gather-daemonset-629q6" Sep 09 23:25:57.141257 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.141178 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d54c2ead-5bf7-48d4-baa8-a00db0adfb86-lib-modules\") pod \"perf-node-gather-daemonset-629q6\" (UID: \"d54c2ead-5bf7-48d4-baa8-a00db0adfb86\") " pod="openshift-must-gather-njqsq/perf-node-gather-daemonset-629q6" Sep 09 23:25:57.141257 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.141208 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d54c2ead-5bf7-48d4-baa8-a00db0adfb86-sys\") pod \"perf-node-gather-daemonset-629q6\" (UID: \"d54c2ead-5bf7-48d4-baa8-a00db0adfb86\") " pod="openshift-must-gather-njqsq/perf-node-gather-daemonset-629q6" Sep 09 23:25:57.242201 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.242168 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d54c2ead-5bf7-48d4-baa8-a00db0adfb86-proc\") pod \"perf-node-gather-daemonset-629q6\" (UID: \"d54c2ead-5bf7-48d4-baa8-a00db0adfb86\") " pod="openshift-must-gather-njqsq/perf-node-gather-daemonset-629q6" Sep 09 23:25:57.242334 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.242203 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d54c2ead-5bf7-48d4-baa8-a00db0adfb86-podres\") pod \"perf-node-gather-daemonset-629q6\" (UID: \"d54c2ead-5bf7-48d4-baa8-a00db0adfb86\") " pod="openshift-must-gather-njqsq/perf-node-gather-daemonset-629q6" Sep 09 23:25:57.242334 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.242235 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr9s6\" (UniqueName: \"kubernetes.io/projected/d54c2ead-5bf7-48d4-baa8-a00db0adfb86-kube-api-access-jr9s6\") pod \"perf-node-gather-daemonset-629q6\" (UID: \"d54c2ead-5bf7-48d4-baa8-a00db0adfb86\") " pod="openshift-must-gather-njqsq/perf-node-gather-daemonset-629q6" Sep 09 23:25:57.242334 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.242301 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d54c2ead-5bf7-48d4-baa8-a00db0adfb86-proc\") pod \"perf-node-gather-daemonset-629q6\" (UID: \"d54c2ead-5bf7-48d4-baa8-a00db0adfb86\") " pod="openshift-must-gather-njqsq/perf-node-gather-daemonset-629q6" Sep 09 23:25:57.242499 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.242348 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d54c2ead-5bf7-48d4-baa8-a00db0adfb86-lib-modules\") pod \"perf-node-gather-daemonset-629q6\" (UID: \"d54c2ead-5bf7-48d4-baa8-a00db0adfb86\") " pod="openshift-must-gather-njqsq/perf-node-gather-daemonset-629q6" Sep 09 23:25:57.242499 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.242378 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d54c2ead-5bf7-48d4-baa8-a00db0adfb86-sys\") pod \"perf-node-gather-daemonset-629q6\" (UID: \"d54c2ead-5bf7-48d4-baa8-a00db0adfb86\") " pod="openshift-must-gather-njqsq/perf-node-gather-daemonset-629q6" Sep 09 23:25:57.242499 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.242350 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d54c2ead-5bf7-48d4-baa8-a00db0adfb86-podres\") pod \"perf-node-gather-daemonset-629q6\" (UID: \"d54c2ead-5bf7-48d4-baa8-a00db0adfb86\") " pod="openshift-must-gather-njqsq/perf-node-gather-daemonset-629q6" Sep 09 23:25:57.242499 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.242426 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d54c2ead-5bf7-48d4-baa8-a00db0adfb86-lib-modules\") pod \"perf-node-gather-daemonset-629q6\" (UID: \"d54c2ead-5bf7-48d4-baa8-a00db0adfb86\") " pod="openshift-must-gather-njqsq/perf-node-gather-daemonset-629q6" Sep 09 23:25:57.242499 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.242466 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d54c2ead-5bf7-48d4-baa8-a00db0adfb86-sys\") pod \"perf-node-gather-daemonset-629q6\" (UID: \"d54c2ead-5bf7-48d4-baa8-a00db0adfb86\") " pod="openshift-must-gather-njqsq/perf-node-gather-daemonset-629q6" Sep 09 23:25:57.249722 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.249696 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr9s6\" (UniqueName: \"kubernetes.io/projected/d54c2ead-5bf7-48d4-baa8-a00db0adfb86-kube-api-access-jr9s6\") pod \"perf-node-gather-daemonset-629q6\" (UID: \"d54c2ead-5bf7-48d4-baa8-a00db0adfb86\") " pod="openshift-must-gather-njqsq/perf-node-gather-daemonset-629q6" Sep 09 23:25:57.296836 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.296810 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-rzk8q_72f66107-ac74-49f5-9782-583434006631/volume-data-source-validator/0.log" Sep 09 23:25:57.379158 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.379109 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-njqsq/perf-node-gather-daemonset-629q6" Sep 09 23:25:57.649546 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.649451 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-njqsq/perf-node-gather-daemonset-629q6"] Sep 09 23:25:57.657232 ip-10-0-143-192 kubenswrapper[2561]: W0909 23:25:57.657206 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd54c2ead_5bf7_48d4_baa8_a00db0adfb86.slice/crio-f9b1c6dad2d2313504f7e33a7acc1db1cb94ee08bcc3e173297dff1c777f9193 WatchSource:0}: Error finding container f9b1c6dad2d2313504f7e33a7acc1db1cb94ee08bcc3e173297dff1c777f9193: Status 404 returned error can't find the container with id f9b1c6dad2d2313504f7e33a7acc1db1cb94ee08bcc3e173297dff1c777f9193 Sep 09 23:25:57.942676 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.942643 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-njqsq/perf-node-gather-daemonset-629q6" event={"ID":"d54c2ead-5bf7-48d4-baa8-a00db0adfb86","Type":"ContainerStarted","Data":"ed3e9e20de93a0eb94e1bf16a88949072d034c5b8568733e251a0f20847dd706"} Sep 09 23:25:57.942840 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.942688 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-njqsq/perf-node-gather-daemonset-629q6" event={"ID":"d54c2ead-5bf7-48d4-baa8-a00db0adfb86","Type":"ContainerStarted","Data":"f9b1c6dad2d2313504f7e33a7acc1db1cb94ee08bcc3e173297dff1c777f9193"} Sep 09 23:25:57.942840 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.942724 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-njqsq/perf-node-gather-daemonset-629q6" Sep 09 23:25:57.959026 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:57.958981 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-njqsq/perf-node-gather-daemonset-629q6" podStartSLOduration=0.958964266 podStartE2EDuration="958.964266ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:25:57.956456493 +0000 UTC m=+1146.642239911" watchObservedRunningTime="2026-09-09 23:25:57.958964266 +0000 UTC m=+1146.644747670" Sep 09 23:25:58.553078 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:58.553045 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tdvkg_e9be36cc-dc4f-414f-98cb-ce27d9d70ff2/dns/0.log" Sep 09 23:25:58.573056 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:58.573024 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tdvkg_e9be36cc-dc4f-414f-98cb-ce27d9d70ff2/kube-rbac-proxy/0.log" Sep 09 23:25:58.615526 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:58.615499 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jqxbd_a843f1d8-0cd1-4396-8fc1-5a22cd20f6e8/dns-node-resolver/0.log" Sep 09 23:25:58.803774 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:58.803695 2561 scope.go:117] "RemoveContainer" containerID="bca67ebbb21c9195e26494183dbf2b704229cf1e45c09e47ab8a3a95506cb6eb" Sep 09 23:25:58.803972 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:25:58.803896 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:25:59.156108 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:59.156039 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d76sb_d702c7a3-03a5-4b8b-9a7f-df843575cc43/node-ca/0.log" Sep 09 23:25:59.804123 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:25:59.804098 2561 scope.go:117] "RemoveContainer" containerID="575ffaae2f5075dde2eb85100cf8f6a1b1a00a177163bfeb735cb37448293405" Sep 09 23:25:59.804500 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:25:59.804251 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:26:00.086826 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:00.086751 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-jnwpf_5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae/discovery/0.log" Sep 09 23:26:00.171304 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:00.171276 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-h555r_eb8317f8-8b9c-4309-94da-ae5b484e14ab/payload-pre-processing/0.log" Sep 09 23:26:00.214170 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:00.214150 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b46465886-mcs6c_6fa8702e-a63e-43aa-907c-e411e4af1b11/router/0.log" Sep 09 23:26:00.804666 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:00.804638 2561 scope.go:117] "RemoveContainer" containerID="8f46ce3935c3b9dee894d62de478a13ebfa89401c11f98949b1061cde719cb9c" Sep 09 23:26:00.805051 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:26:00.804792 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-678lf_llm(ecff2538-a36b-41ca-8484-c5f54bb2cc89)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-678lf" podUID="ecff2538-a36b-41ca-8484-c5f54bb2cc89" Sep 09 23:26:00.849287 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:00.849264 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xbplg_e060ec1e-b4e7-4369-9387-b6817a2bcd4a/serve-healthcheck-canary/0.log" Sep 09 23:26:01.357986 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:01.357956 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p2rqm_8b88ce14-8ff7-4f71-a165-62ccb17063ea/kube-rbac-proxy/0.log" Sep 09 23:26:01.376008 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:01.375985 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p2rqm_8b88ce14-8ff7-4f71-a165-62ccb17063ea/exporter/0.log" Sep 09 23:26:01.395544 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:01.395524 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p2rqm_8b88ce14-8ff7-4f71-a165-62ccb17063ea/extractor/0.log" Sep 09 23:26:03.620212 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:03.620179 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-558d5b9f86-xpn66_e8e5f1bf-f57f-4965-bc29-43e217345fcd/manager/0.log" Sep 09 23:26:03.720369 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:03.720344 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c5b764bdf-dxhww_b3581d49-a5b5-4b09-96aa-9f58bcc64c9b/manager/0.log" Sep 09 23:26:03.738098 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:03.738068 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c5b764bdf-dxhww_b3581d49-a5b5-4b09-96aa-9f58bcc64c9b/copy-manifests/0.log" Sep 09 23:26:03.957441 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:03.957416 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-njqsq/perf-node-gather-daemonset-629q6" Sep 09 23:26:04.920830 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:04.920801 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-66c75bc564-f5ckb_a00d2f07-169b-4bbd-9872-9cd3dd60d69f/manager/0.log" Sep 09 23:26:06.665248 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:06.665165 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-jnwpf_5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae/discovery/0.log" Sep 09 23:26:06.737784 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:06.737753 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-h555r_eb8317f8-8b9c-4309-94da-ae5b484e14ab/payload-pre-processing/0.log" Sep 09 23:26:06.770996 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:06.770978 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b46465886-mcs6c_6fa8702e-a63e-43aa-907c-e411e4af1b11/router/0.log" Sep 09 23:26:07.922048 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:07.922011 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-jnwpf_5a431b8e-ee49-45e1-aa0b-5f67d97ff7ae/discovery/0.log" Sep 09 23:26:07.987911 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:07.987887 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-h555r_eb8317f8-8b9c-4309-94da-ae5b484e14ab/payload-pre-processing/0.log" Sep 09 23:26:08.020802 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:08.020779 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b46465886-mcs6c_6fa8702e-a63e-43aa-907c-e411e4af1b11/router/0.log" Sep 09 23:26:11.698872 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:11.698827 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zncr7_b0c7b946-6758-4433-8081-5e3ea05b3639/kube-multus-additional-cni-plugins/0.log" Sep 09 23:26:11.718675 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:11.718646 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zncr7_b0c7b946-6758-4433-8081-5e3ea05b3639/egress-router-binary-copy/0.log" Sep 09 23:26:11.737438 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:11.737421 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zncr7_b0c7b946-6758-4433-8081-5e3ea05b3639/cni-plugins/0.log" Sep 09 23:26:11.754715 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:11.754694 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zncr7_b0c7b946-6758-4433-8081-5e3ea05b3639/bond-cni-plugin/0.log" Sep 09 23:26:11.773052 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:11.773035 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zncr7_b0c7b946-6758-4433-8081-5e3ea05b3639/routeoverride-cni/0.log" Sep 09 23:26:11.791946 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:11.791930 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zncr7_b0c7b946-6758-4433-8081-5e3ea05b3639/whereabouts-cni-bincopy/0.log" Sep 09 23:26:11.810524 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:11.810508 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zncr7_b0c7b946-6758-4433-8081-5e3ea05b3639/whereabouts-cni/0.log" Sep 09 23:26:11.870590 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:11.870572 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-t69sp_1c6b309f-307c-4a0f-a0d1-586fadcd13b1/kube-multus/0.log" Sep 09 23:26:11.925947 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:11.925925 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-j89rl_9e224f1b-9ce8-484e-8de5-f4f4f49a5eab/network-metrics-daemon/0.log" Sep 09 23:26:11.942596 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:11.942578 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-j89rl_9e224f1b-9ce8-484e-8de5-f4f4f49a5eab/kube-rbac-proxy/0.log" Sep 09 23:26:12.803978 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:12.803955 2561 scope.go:117] "RemoveContainer" containerID="bca67ebbb21c9195e26494183dbf2b704229cf1e45c09e47ab8a3a95506cb6eb" Sep 09 23:26:12.804387 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:12.804099 2561 scope.go:117] "RemoveContainer" containerID="575ffaae2f5075dde2eb85100cf8f6a1b1a00a177163bfeb735cb37448293405" Sep 09 23:26:12.804387 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:26:12.804172 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8tlsl_llm(bbc27533-85d7-4f55-8f45-dcd10685b8f6)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8tlsl" podUID="bbc27533-85d7-4f55-8f45-dcd10685b8f6" Sep 09 23:26:12.804387 ip-10-0-143-192 kubenswrapper[2561]: E0909 23:26:12.804268 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j_llm(2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-tpf2j" podUID="2ddcfdf7-7daf-4b30-a3cf-18d5fe84c4b9" Sep 09 23:26:12.945034 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:12.945008 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbrgn_6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21/ovn-controller/0.log" Sep 09 23:26:12.961379 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:12.961359 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbrgn_6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21/ovn-acl-logging/0.log" Sep 09 23:26:12.985136 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:12.985113 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbrgn_6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21/kube-rbac-proxy-node/0.log" Sep 09 23:26:13.003333 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:13.003279 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbrgn_6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21/kube-rbac-proxy-ovn-metrics/0.log" Sep 09 23:26:13.021476 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:13.021457 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbrgn_6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21/northd/0.log" Sep 09 23:26:13.041201 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:13.041186 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbrgn_6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21/nbdb/0.log" Sep 09 23:26:13.059201 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:13.059179 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbrgn_6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21/sbdb/0.log" Sep 09 23:26:13.096458 ip-10-0-143-192 kubenswrapper[2561]: I0909 23:26:13.096443 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pbrgn_6ed8aca2-ce39-4ccd-8dfc-ef9412d0fb21/ovnkube-controller/0.log"