Jul 10 20:32:40.712826 ip-10-0-139-64 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 10 20:32:40.712838 ip-10-0-139-64 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 10 20:32:40.712847 ip-10-0-139-64 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 10 20:32:40.713177 ip-10-0-139-64 systemd[1]: Failed to start Kubernetes Kubelet. Jul 10 20:32:50.752862 ip-10-0-139-64 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 10 20:32:50.752878 ip-10-0-139-64 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 2986e33d2e8047f98f959172f575f9b2 -- Jul 10 20:34:47.101157 ip-10-0-139-64 systemd[1]: Starting Kubernetes Kubelet... Jul 10 20:34:47.532262 ip-10-0-139-64 kubenswrapper[2530]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 20:34:47.532262 ip-10-0-139-64 kubenswrapper[2530]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 20:34:47.532262 ip-10-0-139-64 kubenswrapper[2530]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 20:34:47.532262 ip-10-0-139-64 kubenswrapper[2530]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 20:34:47.532262 ip-10-0-139-64 kubenswrapper[2530]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 20:34:47.533554 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.533451 2530 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 20:34:47.539979 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.539953 2530 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 20:34:47.539979 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.539973 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 20:34:47.539979 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.539979 2530 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 20:34:47.539979 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.539983 2530 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 20:34:47.539979 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.539987 2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 20:34:47.540304 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.539991 2530 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 20:34:47.540304 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.539996 2530 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 20:34:47.540304 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540002 2530 feature_gate.go:328] unrecognized feature gate: Example Jul 10 20:34:47.540304 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540005 2530 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 20:34:47.540304 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540009 2530 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 20:34:47.540304 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540013 2530 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 20:34:47.540304 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540017 2530 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 20:34:47.540304 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540022 2530 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 20:34:47.540304 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540025 2530 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 20:34:47.540304 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540029 2530 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 20:34:47.540304 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540033 2530 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 20:34:47.540304 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540039 2530 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 20:34:47.540304 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540044 2530 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 20:34:47.540304 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540048 2530 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 20:34:47.540304 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540052 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 20:34:47.540304 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540056 2530 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 20:34:47.540304 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540060 2530 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 20:34:47.540304 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540064 2530 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 20:34:47.540304 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540069 2530 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 20:34:47.541137 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540073 2530 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 20:34:47.541137 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540077 2530 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 20:34:47.541137 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540081 2530 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 20:34:47.541137 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540087 2530 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 20:34:47.541137 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540091 2530 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 20:34:47.541137 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540095 2530 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 20:34:47.541137 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540100 2530 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 20:34:47.541137 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540105 2530 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 20:34:47.541137 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540109 2530 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 20:34:47.541137 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540113 2530 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 20:34:47.541137 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540117 2530 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 20:34:47.541137 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540121 2530 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 20:34:47.541137 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540126 2530 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 20:34:47.541137 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540130 2530 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 20:34:47.541137 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540136 2530 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 20:34:47.541137 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540141 2530 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 20:34:47.541137 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540146 2530 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 20:34:47.541137 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540150 2530 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 20:34:47.541137 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540157 2530 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 20:34:47.541137 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540162 2530 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 20:34:47.541725 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540166 2530 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 20:34:47.541725 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540171 2530 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 20:34:47.541725 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540175 2530 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 20:34:47.541725 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540183 2530 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 20:34:47.541725 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540190 2530 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 20:34:47.541725 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540197 2530 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 20:34:47.541725 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540203 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 20:34:47.541725 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540208 2530 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 20:34:47.541725 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540213 2530 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 20:34:47.541725 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540218 2530 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 20:34:47.541725 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540222 2530 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 20:34:47.541725 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540227 2530 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 20:34:47.541725 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540232 2530 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 20:34:47.541725 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540237 2530 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 20:34:47.541725 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540242 2530 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 20:34:47.541725 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540246 2530 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 20:34:47.541725 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540251 2530 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 20:34:47.541725 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540255 2530 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 20:34:47.541725 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540260 2530 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 20:34:47.541725 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540265 2530 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 20:34:47.542302 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540270 2530 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 20:34:47.542302 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540274 2530 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 20:34:47.542302 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540279 2530 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 20:34:47.542302 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540283 2530 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 20:34:47.542302 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540288 2530 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 20:34:47.542302 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540293 2530 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 20:34:47.542302 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540297 2530 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 20:34:47.542302 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540302 2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 20:34:47.542302 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540307 2530 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 20:34:47.542302 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540311 2530 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 20:34:47.542302 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540316 2530 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 20:34:47.542302 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540322 2530 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 20:34:47.542302 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540326 2530 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 20:34:47.542302 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540331 2530 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 20:34:47.542302 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540335 2530 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 20:34:47.542302 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540340 2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 20:34:47.542302 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540344 2530 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 20:34:47.542302 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540349 2530 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 20:34:47.542302 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540354 2530 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 20:34:47.542302 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540358 2530 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 20:34:47.542825 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540363 2530 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 20:34:47.542825 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540368 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 20:34:47.542825 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540378 2530 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 20:34:47.542825 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540385 2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 20:34:47.542825 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540391 2530 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 20:34:47.542825 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540396 2530 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 20:34:47.542825 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540401 2530 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 20:34:47.542825 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540405 2530 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 20:34:47.542825 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540410 2530 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 20:34:47.542825 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540414 2530 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 20:34:47.542825 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540422 2530 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 20:34:47.542825 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540429 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 20:34:47.542825 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540435 2530 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 20:34:47.542825 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540440 2530 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 20:34:47.542825 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.540445 2530 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 20:34:47.543484 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543421 2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 20:34:47.543484 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543434 2530 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 20:34:47.543484 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543440 2530 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 20:34:47.543484 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543445 2530 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 20:34:47.543484 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543451 2530 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 20:34:47.543484 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543456 2530 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 20:34:47.543484 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543461 2530 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 20:34:47.543484 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543465 2530 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 20:34:47.543484 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543470 2530 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 20:34:47.543484 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543475 2530 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 20:34:47.543484 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543480 2530 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 20:34:47.544006 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543505 2530 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 20:34:47.544006 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543513 2530 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 20:34:47.544006 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543518 2530 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 20:34:47.544006 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543523 2530 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 20:34:47.544006 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543529 2530 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 20:34:47.544006 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543534 2530 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 20:34:47.544006 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543538 2530 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 20:34:47.544006 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543543 2530 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 20:34:47.544006 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543547 2530 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 20:34:47.544006 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543552 2530 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 20:34:47.544006 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543556 2530 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 20:34:47.544006 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543561 2530 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 20:34:47.544006 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543565 2530 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 20:34:47.544006 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543570 2530 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 20:34:47.544006 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543575 2530 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 20:34:47.544006 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543579 2530 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 20:34:47.544006 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543584 2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 20:34:47.544006 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543588 2530 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 20:34:47.544006 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543593 2530 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 20:34:47.544006 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543597 2530 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 20:34:47.544896 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543602 2530 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 20:34:47.544896 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543606 2530 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 20:34:47.544896 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543613 2530 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 20:34:47.544896 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543620 2530 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 20:34:47.544896 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543625 2530 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 20:34:47.544896 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543629 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 20:34:47.544896 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543634 2530 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 20:34:47.544896 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543638 2530 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 20:34:47.544896 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543643 2530 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 20:34:47.544896 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543650 2530 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 20:34:47.544896 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543656 2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 20:34:47.544896 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543662 2530 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 20:34:47.544896 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543667 2530 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 20:34:47.544896 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543674 2530 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 20:34:47.544896 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543679 2530 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 20:34:47.544896 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543684 2530 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 20:34:47.544896 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543689 2530 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 20:34:47.544896 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543695 2530 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 20:34:47.544896 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543700 2530 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 20:34:47.545803 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543705 2530 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 20:34:47.545803 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543710 2530 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 20:34:47.545803 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543717 2530 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 20:34:47.545803 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543722 2530 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 20:34:47.545803 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543727 2530 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 20:34:47.545803 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543732 2530 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 20:34:47.545803 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543737 2530 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 20:34:47.545803 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543743 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 20:34:47.545803 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543748 2530 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 20:34:47.545803 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543753 2530 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 20:34:47.545803 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543758 2530 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 20:34:47.545803 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543762 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 20:34:47.545803 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543767 2530 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 20:34:47.545803 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543771 2530 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 20:34:47.545803 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543776 2530 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 20:34:47.545803 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543780 2530 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 20:34:47.545803 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543785 2530 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 20:34:47.545803 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543790 2530 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 20:34:47.545803 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543794 2530 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 20:34:47.545803 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543800 2530 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 20:34:47.546551 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543804 2530 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 20:34:47.546551 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543809 2530 feature_gate.go:328] unrecognized feature gate: Example Jul 10 20:34:47.546551 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543813 2530 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 20:34:47.546551 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543818 2530 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 20:34:47.546551 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543823 2530 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 20:34:47.546551 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543830 2530 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 20:34:47.546551 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543836 2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 20:34:47.546551 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543840 2530 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 20:34:47.546551 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543845 2530 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 20:34:47.546551 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543850 2530 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 20:34:47.546551 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543854 2530 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 20:34:47.546551 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543859 2530 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 20:34:47.546551 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543864 2530 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 20:34:47.546551 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543868 2530 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 20:34:47.546551 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543873 2530 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 20:34:47.546551 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543880 2530 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 20:34:47.546551 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543886 2530 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 20:34:47.546551 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543891 2530 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 20:34:47.546551 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543896 2530 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 20:34:47.547079 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543901 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 20:34:47.547079 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543923 2530 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 20:34:47.547079 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543929 2530 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 20:34:47.547079 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543934 2530 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 20:34:47.547079 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543939 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 20:34:47.547079 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543943 2530 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 20:34:47.547079 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543948 2530 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 20:34:47.547079 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543953 2530 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 20:34:47.547079 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543957 2530 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 20:34:47.547079 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.543962 2530 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 20:34:47.547079 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544640 2530 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 20:34:47.547079 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544653 2530 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 20:34:47.547079 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544660 2530 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 20:34:47.547079 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544666 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 20:34:47.547079 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544671 2530 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 20:34:47.547079 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544677 2530 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 20:34:47.547079 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544682 2530 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 20:34:47.547079 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544688 2530 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 20:34:47.547079 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544693 2530 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 20:34:47.547079 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544699 2530 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 20:34:47.547613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544704 2530 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 20:34:47.547613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544709 2530 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 20:34:47.547613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544714 2530 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 20:34:47.547613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544719 2530 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 20:34:47.547613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544724 2530 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 20:34:47.547613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544729 2530 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 20:34:47.547613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544733 2530 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 20:34:47.547613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544738 2530 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 20:34:47.547613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544743 2530 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 20:34:47.547613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544747 2530 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 20:34:47.547613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544752 2530 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 20:34:47.547613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544757 2530 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 20:34:47.547613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544761 2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 20:34:47.547613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544766 2530 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 20:34:47.547613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544771 2530 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 20:34:47.547613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544775 2530 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 20:34:47.547613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544780 2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 20:34:47.547613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544786 2530 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 20:34:47.547613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544791 2530 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 20:34:47.547613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544795 2530 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 20:34:47.548142 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544800 2530 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 20:34:47.548142 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544805 2530 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 20:34:47.548142 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544810 2530 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 20:34:47.548142 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544815 2530 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 20:34:47.548142 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544822 2530 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 20:34:47.548142 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544828 2530 feature_gate.go:328] unrecognized feature gate: Example Jul 10 20:34:47.548142 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544833 2530 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 20:34:47.548142 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544838 2530 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 20:34:47.548142 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544843 2530 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 20:34:47.548142 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544848 2530 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 20:34:47.548142 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544852 2530 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 20:34:47.548142 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544858 2530 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 20:34:47.548142 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544863 2530 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 20:34:47.548142 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544868 2530 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 20:34:47.548142 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544872 2530 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 20:34:47.548142 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544877 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 20:34:47.548142 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544881 2530 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 20:34:47.548142 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544886 2530 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 20:34:47.548142 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544890 2530 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 20:34:47.548142 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544894 2530 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 20:34:47.548688 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544899 2530 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 20:34:47.548688 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544903 2530 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 20:34:47.548688 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544928 2530 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 20:34:47.548688 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544933 2530 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 20:34:47.548688 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544937 2530 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 20:34:47.548688 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544942 2530 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 20:34:47.548688 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544946 2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 20:34:47.548688 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544950 2530 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 20:34:47.548688 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544954 2530 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 20:34:47.548688 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544959 2530 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 20:34:47.548688 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544965 2530 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 20:34:47.548688 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544970 2530 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 20:34:47.548688 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544975 2530 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 20:34:47.548688 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544980 2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 20:34:47.548688 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544985 2530 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 20:34:47.548688 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544990 2530 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 20:34:47.548688 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.544996 2530 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 20:34:47.548688 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545001 2530 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 20:34:47.548688 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545006 2530 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 20:34:47.548688 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545012 2530 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 20:34:47.549237 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545017 2530 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 20:34:47.549237 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545022 2530 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 20:34:47.549237 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545026 2530 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 20:34:47.549237 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545031 2530 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 20:34:47.549237 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545037 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 20:34:47.549237 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545042 2530 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 20:34:47.549237 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545046 2530 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 20:34:47.549237 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545051 2530 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 20:34:47.549237 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545055 2530 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 20:34:47.549237 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545060 2530 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 20:34:47.549237 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545065 2530 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 20:34:47.549237 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545070 2530 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 20:34:47.549237 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545074 2530 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 20:34:47.549237 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545079 2530 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 20:34:47.549237 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545083 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 20:34:47.549237 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545088 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 20:34:47.549237 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545092 2530 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 20:34:47.549237 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545097 2530 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 20:34:47.549237 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545101 2530 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 20:34:47.549237 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545106 2530 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 20:34:47.549739 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545111 2530 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 20:34:47.549739 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545116 2530 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 20:34:47.549739 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545121 2530 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 20:34:47.549739 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545126 2530 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 20:34:47.549739 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545130 2530 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 20:34:47.549739 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545135 2530 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 20:34:47.549739 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545140 2530 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 20:34:47.549739 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545144 2530 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 20:34:47.549739 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.545149 2530 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 20:34:47.549739 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545258 2530 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 20:34:47.549739 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545271 2530 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 20:34:47.549739 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545283 2530 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 20:34:47.549739 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545291 2530 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 20:34:47.549739 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545298 2530 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 20:34:47.549739 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545303 2530 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 20:34:47.549739 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545311 2530 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 20:34:47.549739 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545319 2530 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 20:34:47.549739 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545331 2530 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 20:34:47.549739 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545339 2530 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 20:34:47.549739 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545349 2530 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 20:34:47.549739 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545360 2530 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 20:34:47.549739 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545369 2530 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 20:34:47.550395 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545374 2530 flags.go:64] FLAG: --cgroup-root="" Jul 10 20:34:47.550395 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545379 2530 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 20:34:47.550395 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545384 2530 flags.go:64] FLAG: --client-ca-file="" Jul 10 20:34:47.550395 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545390 2530 flags.go:64] FLAG: --cloud-provider="external" Jul 10 20:34:47.550395 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545395 2530 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 20:34:47.550395 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545403 2530 flags.go:64] FLAG: --cluster-domain="" Jul 10 20:34:47.550395 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545408 2530 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 20:34:47.550395 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545413 2530 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 20:34:47.550395 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545422 2530 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 20:34:47.550395 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545428 2530 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 20:34:47.550395 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545434 2530 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 20:34:47.550395 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545440 2530 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 20:34:47.550395 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545445 2530 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 20:34:47.550395 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545451 2530 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 20:34:47.550395 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545457 2530 flags.go:64] FLAG: --contention-profiling="false" Jul 10 20:34:47.550395 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545462 2530 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 20:34:47.550395 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545467 2530 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 20:34:47.550395 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545473 2530 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 20:34:47.550395 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545478 2530 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 20:34:47.550395 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545485 2530 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 20:34:47.550395 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545491 2530 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 20:34:47.550395 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545496 2530 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 20:34:47.550395 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545501 2530 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 20:34:47.550395 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545506 2530 flags.go:64] FLAG: --enable-server="true" Jul 10 20:34:47.550395 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545511 2530 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 20:34:47.551042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545521 2530 flags.go:64] FLAG: --event-burst="100" Jul 10 20:34:47.551042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545526 2530 flags.go:64] FLAG: --event-qps="50" Jul 10 20:34:47.551042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545531 2530 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 20:34:47.551042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545538 2530 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 20:34:47.551042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545544 2530 flags.go:64] FLAG: --eviction-hard="" Jul 10 20:34:47.551042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545550 2530 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 20:34:47.551042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545555 2530 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 20:34:47.551042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545576 2530 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 20:34:47.551042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545581 2530 flags.go:64] FLAG: --eviction-soft="" Jul 10 20:34:47.551042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545586 2530 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 20:34:47.551042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545591 2530 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 20:34:47.551042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545597 2530 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 20:34:47.551042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545602 2530 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 20:34:47.551042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545607 2530 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 20:34:47.551042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545612 2530 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 20:34:47.551042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545619 2530 flags.go:64] FLAG: --feature-gates="" Jul 10 20:34:47.551042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545626 2530 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 20:34:47.551042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545631 2530 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 20:34:47.551042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545639 2530 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 20:34:47.551042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545645 2530 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 20:34:47.551042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545650 2530 flags.go:64] FLAG: --healthz-port="10248" Jul 10 20:34:47.551042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545655 2530 flags.go:64] FLAG: --help="false" Jul 10 20:34:47.551042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545660 2530 flags.go:64] FLAG: --hostname-override="ip-10-0-139-64.ec2.internal" Jul 10 20:34:47.551042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545666 2530 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 20:34:47.551042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545671 2530 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 20:34:47.551833 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545677 2530 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 20:34:47.551833 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545683 2530 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 20:34:47.551833 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545690 2530 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 20:34:47.551833 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545695 2530 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 20:34:47.551833 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545699 2530 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 20:34:47.551833 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545705 2530 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 20:34:47.551833 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545710 2530 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 20:34:47.551833 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545715 2530 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 20:34:47.551833 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545721 2530 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 20:34:47.551833 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545726 2530 flags.go:64] FLAG: --kube-reserved="" Jul 10 20:34:47.551833 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545731 2530 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 20:34:47.551833 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545737 2530 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 20:34:47.551833 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545743 2530 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 20:34:47.551833 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545748 2530 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 20:34:47.551833 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545754 2530 flags.go:64] FLAG: --lock-file="" Jul 10 20:34:47.551833 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545759 2530 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 20:34:47.551833 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545764 2530 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 20:34:47.551833 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545770 2530 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 20:34:47.551833 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545780 2530 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 20:34:47.551833 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545785 2530 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 20:34:47.551833 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545790 2530 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 20:34:47.551833 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545795 2530 flags.go:64] FLAG: --logging-format="text" Jul 10 20:34:47.551833 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545804 2530 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 20:34:47.552480 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545809 2530 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 20:34:47.552480 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545815 2530 flags.go:64] FLAG: --manifest-url="" Jul 10 20:34:47.552480 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545823 2530 flags.go:64] FLAG: --manifest-url-header="" Jul 10 20:34:47.552480 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545830 2530 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 20:34:47.552480 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545836 2530 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 20:34:47.552480 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545844 2530 flags.go:64] FLAG: --max-pods="110" Jul 10 20:34:47.552480 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545849 2530 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 20:34:47.552480 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545855 2530 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 20:34:47.552480 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545860 2530 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 20:34:47.552480 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545865 2530 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 20:34:47.552480 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545871 2530 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 20:34:47.552480 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545876 2530 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 20:34:47.552480 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545881 2530 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 20:34:47.552480 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545895 2530 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 20:34:47.552480 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545900 2530 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 20:34:47.552480 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545925 2530 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 20:34:47.552480 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545931 2530 flags.go:64] FLAG: --pod-cidr="" Jul 10 20:34:47.552480 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545937 2530 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 20:34:47.552480 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545946 2530 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 20:34:47.552480 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545951 2530 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 20:34:47.552480 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545957 2530 flags.go:64] FLAG: --pods-per-core="0" Jul 10 20:34:47.552480 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545964 2530 flags.go:64] FLAG: --port="10250" Jul 10 20:34:47.552480 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545970 2530 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 20:34:47.552480 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545975 2530 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a2b7e3f232a94bba" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545981 2530 flags.go:64] FLAG: --qos-reserved="" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545987 2530 flags.go:64] FLAG: --read-only-port="10255" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545992 2530 flags.go:64] FLAG: --register-node="true" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.545997 2530 flags.go:64] FLAG: --register-with-taints="" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546004 2530 flags.go:64] FLAG: --registry-burst="10" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546008 2530 flags.go:64] FLAG: --registry-qps="5" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546014 2530 flags.go:64] FLAG: --reserved-cpus="" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546022 2530 flags.go:64] FLAG: --reserved-memory="" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546028 2530 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546034 2530 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546039 2530 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546046 2530 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546052 2530 flags.go:64] FLAG: --runonce="false" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546057 2530 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546063 2530 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546069 2530 flags.go:64] FLAG: --seccomp-default="false" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546074 2530 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546079 2530 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546085 2530 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546090 2530 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546096 2530 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546101 2530 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546107 2530 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546112 2530 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546117 2530 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 20:34:47.553128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546122 2530 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 20:34:47.553798 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546127 2530 flags.go:64] FLAG: --system-cgroups="" Jul 10 20:34:47.553798 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546132 2530 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 20:34:47.553798 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546142 2530 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 20:34:47.553798 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546148 2530 flags.go:64] FLAG: --tls-cert-file="" Jul 10 20:34:47.553798 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546154 2530 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 20:34:47.553798 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546161 2530 flags.go:64] FLAG: --tls-min-version="" Jul 10 20:34:47.553798 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546166 2530 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 20:34:47.553798 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546171 2530 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 20:34:47.553798 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546176 2530 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 20:34:47.553798 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546182 2530 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 20:34:47.553798 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546188 2530 flags.go:64] FLAG: --v="2" Jul 10 20:34:47.553798 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546195 2530 flags.go:64] FLAG: --version="false" Jul 10 20:34:47.553798 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546202 2530 flags.go:64] FLAG: --vmodule="" Jul 10 20:34:47.553798 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546211 2530 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 20:34:47.553798 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546217 2530 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 20:34:47.553798 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546446 2530 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 20:34:47.553798 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546464 2530 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 20:34:47.553798 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546468 2530 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 20:34:47.553798 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546472 2530 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 20:34:47.553798 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546476 2530 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 20:34:47.553798 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546479 2530 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 20:34:47.553798 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546483 2530 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 20:34:47.553798 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546486 2530 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 20:34:47.554466 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546489 2530 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 20:34:47.554466 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546492 2530 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 20:34:47.554466 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546495 2530 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 20:34:47.554466 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546498 2530 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 20:34:47.554466 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546501 2530 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 20:34:47.554466 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546504 2530 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 20:34:47.554466 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546507 2530 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 20:34:47.554466 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546512 2530 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 20:34:47.554466 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546516 2530 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 20:34:47.554466 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546519 2530 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 20:34:47.554466 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546522 2530 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 20:34:47.554466 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546525 2530 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 20:34:47.554466 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546528 2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 20:34:47.554466 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546531 2530 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 20:34:47.554466 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546534 2530 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 20:34:47.554466 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546537 2530 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 20:34:47.554466 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546539 2530 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 20:34:47.554466 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546542 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 20:34:47.554466 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546545 2530 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 20:34:47.554466 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546548 2530 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 20:34:47.555039 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546551 2530 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 20:34:47.555039 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546554 2530 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 20:34:47.555039 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546557 2530 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 20:34:47.555039 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546560 2530 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 20:34:47.555039 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546563 2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 20:34:47.555039 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546577 2530 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 20:34:47.555039 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546580 2530 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 20:34:47.555039 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546583 2530 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 20:34:47.555039 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546586 2530 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 20:34:47.555039 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546589 2530 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 20:34:47.555039 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546592 2530 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 20:34:47.555039 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546595 2530 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 20:34:47.555039 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546597 2530 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 20:34:47.555039 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546600 2530 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 20:34:47.555039 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546603 2530 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 20:34:47.555039 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546606 2530 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 20:34:47.555039 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546609 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 20:34:47.555039 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546611 2530 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 20:34:47.555039 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546614 2530 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 20:34:47.555039 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546617 2530 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 20:34:47.555570 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546620 2530 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 20:34:47.555570 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546622 2530 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 20:34:47.555570 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546625 2530 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 20:34:47.555570 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546628 2530 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 20:34:47.555570 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546631 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 20:34:47.555570 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546633 2530 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 20:34:47.555570 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546637 2530 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 20:34:47.555570 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546639 2530 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 20:34:47.555570 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546642 2530 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 20:34:47.555570 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546645 2530 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 20:34:47.555570 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546647 2530 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 20:34:47.555570 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546650 2530 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 20:34:47.555570 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546653 2530 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 20:34:47.555570 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546656 2530 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 20:34:47.555570 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546658 2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 20:34:47.555570 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546662 2530 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 20:34:47.555570 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546665 2530 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 20:34:47.555570 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546669 2530 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 20:34:47.555570 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546678 2530 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 20:34:47.555570 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546681 2530 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 20:34:47.556153 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546684 2530 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 20:34:47.556153 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546687 2530 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 20:34:47.556153 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546692 2530 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 20:34:47.556153 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546697 2530 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 20:34:47.556153 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546700 2530 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 20:34:47.556153 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546703 2530 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 20:34:47.556153 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546705 2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 20:34:47.556153 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546708 2530 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 20:34:47.556153 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546711 2530 feature_gate.go:328] unrecognized feature gate: Example Jul 10 20:34:47.556153 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546714 2530 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 20:34:47.556153 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546718 2530 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 20:34:47.556153 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546722 2530 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 20:34:47.556153 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546725 2530 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 20:34:47.556153 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546727 2530 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 20:34:47.556153 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546730 2530 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 20:34:47.556153 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546733 2530 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 20:34:47.556153 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546736 2530 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 20:34:47.556153 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546739 2530 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 20:34:47.556153 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546741 2530 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 20:34:47.556649 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546744 2530 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 20:34:47.556649 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546747 2530 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 20:34:47.556649 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546750 2530 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 20:34:47.556649 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546753 2530 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 20:34:47.556649 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546756 2530 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 20:34:47.556649 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546759 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 20:34:47.556649 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546762 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 20:34:47.556649 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546764 2530 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 20:34:47.556649 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546768 2530 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 20:34:47.556649 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546771 2530 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 20:34:47.556649 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546775 2530 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 20:34:47.556649 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.546778 2530 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 20:34:47.556649 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.546791 2530 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 20:34:47.556649 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.554144 2530 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 20:34:47.556649 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.554166 2530 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 20:34:47.557084 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.554961 2530 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 20:34:47.557084 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.554981 2530 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 20:34:47.557084 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555054 2530 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 20:34:47.557084 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555060 2530 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 20:34:47.557084 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555065 2530 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 20:34:47.557084 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555068 2530 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 20:34:47.557084 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555071 2530 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 20:34:47.557084 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555074 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 20:34:47.557084 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555077 2530 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 20:34:47.557084 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555080 2530 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 20:34:47.557084 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555083 2530 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 20:34:47.557084 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555086 2530 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 20:34:47.557084 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555089 2530 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 20:34:47.557084 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555093 2530 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 20:34:47.557084 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555096 2530 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 20:34:47.557084 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555099 2530 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 20:34:47.557084 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555101 2530 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 20:34:47.557084 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555104 2530 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 20:34:47.557084 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555107 2530 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 20:34:47.557084 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555111 2530 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 20:34:47.557613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555116 2530 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 20:34:47.557613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555119 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 20:34:47.557613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555122 2530 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 20:34:47.557613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555125 2530 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 20:34:47.557613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555127 2530 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 20:34:47.557613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555130 2530 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 20:34:47.557613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555133 2530 feature_gate.go:328] unrecognized feature gate: Example Jul 10 20:34:47.557613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555136 2530 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 20:34:47.557613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555139 2530 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 20:34:47.557613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555143 2530 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 20:34:47.557613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555146 2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 20:34:47.557613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555149 2530 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 20:34:47.557613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555154 2530 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 20:34:47.557613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555158 2530 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 20:34:47.557613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555161 2530 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 20:34:47.557613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555164 2530 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 20:34:47.557613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555167 2530 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 20:34:47.557613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555172 2530 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 20:34:47.557613 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555176 2530 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 20:34:47.558148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555179 2530 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 20:34:47.558148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555183 2530 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 20:34:47.558148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555186 2530 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 20:34:47.558148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555189 2530 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 20:34:47.558148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555192 2530 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 20:34:47.558148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555196 2530 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 20:34:47.558148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555199 2530 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 20:34:47.558148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555202 2530 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 20:34:47.558148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555205 2530 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 20:34:47.558148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555208 2530 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 20:34:47.558148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555211 2530 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 20:34:47.558148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555214 2530 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 20:34:47.558148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555216 2530 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 20:34:47.558148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555219 2530 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 20:34:47.558148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555222 2530 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 20:34:47.558148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555225 2530 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 20:34:47.558148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555228 2530 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 20:34:47.558148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555231 2530 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 20:34:47.558148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555233 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 20:34:47.558148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555236 2530 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 20:34:47.558679 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555239 2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 20:34:47.558679 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555242 2530 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 20:34:47.558679 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555245 2530 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 20:34:47.558679 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555249 2530 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 20:34:47.558679 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555252 2530 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 20:34:47.558679 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555255 2530 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 20:34:47.558679 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555258 2530 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 20:34:47.558679 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555261 2530 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 20:34:47.558679 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555264 2530 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 20:34:47.558679 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555267 2530 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 20:34:47.558679 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555269 2530 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 20:34:47.558679 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555272 2530 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 20:34:47.558679 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555275 2530 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 20:34:47.558679 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555278 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 20:34:47.558679 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555281 2530 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 20:34:47.558679 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555284 2530 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 20:34:47.558679 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555286 2530 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 20:34:47.558679 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555289 2530 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 20:34:47.558679 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555292 2530 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 20:34:47.558679 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555295 2530 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 20:34:47.559221 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555297 2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 20:34:47.559221 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555300 2530 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 20:34:47.559221 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555303 2530 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 20:34:47.559221 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555306 2530 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 20:34:47.559221 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555308 2530 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 20:34:47.559221 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555311 2530 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 20:34:47.559221 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555314 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 20:34:47.559221 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555316 2530 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 20:34:47.559221 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555319 2530 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 20:34:47.559221 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555322 2530 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 20:34:47.559221 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555325 2530 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 20:34:47.559221 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555328 2530 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 20:34:47.559221 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555331 2530 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 20:34:47.559221 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555333 2530 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 20:34:47.559221 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555336 2530 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 20:34:47.559221 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555340 2530 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 20:34:47.559221 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555343 2530 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 20:34:47.559221 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555347 2530 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 20:34:47.559221 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555350 2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 20:34:47.559740 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555353 2530 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 20:34:47.559740 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555355 2530 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 20:34:47.559740 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555358 2530 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 20:34:47.559740 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.555364 2530 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 20:34:47.559740 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555489 2530 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 20:34:47.559740 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555494 2530 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 20:34:47.559740 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555497 2530 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 20:34:47.559740 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555500 2530 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 20:34:47.559740 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555503 2530 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 20:34:47.559740 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555506 2530 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 20:34:47.559740 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555509 2530 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 20:34:47.559740 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555513 2530 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 20:34:47.559740 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555517 2530 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 20:34:47.559740 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555520 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 20:34:47.560347 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555524 2530 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 20:34:47.560347 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555527 2530 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 20:34:47.560347 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555530 2530 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 20:34:47.560347 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555534 2530 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 20:34:47.560347 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555539 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 20:34:47.560347 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555542 2530 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 20:34:47.560347 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555546 2530 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 20:34:47.560347 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555549 2530 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 20:34:47.560347 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555552 2530 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 20:34:47.560347 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555555 2530 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 20:34:47.560347 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555558 2530 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 20:34:47.560347 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555561 2530 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 20:34:47.560347 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555564 2530 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 20:34:47.560347 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555567 2530 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 20:34:47.560347 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555570 2530 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 20:34:47.560347 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555573 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 20:34:47.560347 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555578 2530 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 20:34:47.560347 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555581 2530 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 20:34:47.560347 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555584 2530 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 20:34:47.561102 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555587 2530 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 20:34:47.561102 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555590 2530 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 20:34:47.561102 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555593 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 20:34:47.561102 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555595 2530 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 20:34:47.561102 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555598 2530 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 20:34:47.561102 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555600 2530 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 20:34:47.561102 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555603 2530 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 20:34:47.561102 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555606 2530 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 20:34:47.561102 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555609 2530 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 20:34:47.561102 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555612 2530 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 20:34:47.561102 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555614 2530 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 20:34:47.561102 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555617 2530 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 20:34:47.561102 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555620 2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 20:34:47.561102 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555623 2530 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 20:34:47.561102 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555626 2530 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 20:34:47.561102 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555628 2530 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 20:34:47.561102 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555631 2530 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 20:34:47.561102 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555634 2530 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 20:34:47.561102 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555637 2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 20:34:47.561606 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555640 2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 20:34:47.561606 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555642 2530 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 20:34:47.561606 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555645 2530 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 20:34:47.561606 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555648 2530 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 20:34:47.561606 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555651 2530 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 20:34:47.561606 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555653 2530 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 20:34:47.561606 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555656 2530 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 20:34:47.561606 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555659 2530 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 20:34:47.561606 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555661 2530 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 20:34:47.561606 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555665 2530 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 20:34:47.561606 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555668 2530 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 20:34:47.561606 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555672 2530 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 20:34:47.561606 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555675 2530 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 20:34:47.561606 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555678 2530 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 20:34:47.561606 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555681 2530 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 20:34:47.561606 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555684 2530 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 20:34:47.561606 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555686 2530 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 20:34:47.561606 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555689 2530 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 20:34:47.561606 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555692 2530 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 20:34:47.561606 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555695 2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 20:34:47.562148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555698 2530 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 20:34:47.562148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555700 2530 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 20:34:47.562148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555703 2530 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 20:34:47.562148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555706 2530 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 20:34:47.562148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555708 2530 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 20:34:47.562148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555711 2530 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 20:34:47.562148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555714 2530 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 20:34:47.562148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555717 2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 20:34:47.562148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555719 2530 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 20:34:47.562148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555722 2530 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 20:34:47.562148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555725 2530 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 20:34:47.562148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555728 2530 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 20:34:47.562148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555730 2530 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 20:34:47.562148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555733 2530 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 20:34:47.562148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555736 2530 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 20:34:47.562148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555739 2530 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 20:34:47.562148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555742 2530 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 20:34:47.562148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555745 2530 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 20:34:47.562148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555747 2530 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 20:34:47.562148 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555750 2530 feature_gate.go:328] unrecognized feature gate: Example Jul 10 20:34:47.562683 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555753 2530 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 20:34:47.562683 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555755 2530 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 20:34:47.562683 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555758 2530 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 20:34:47.562683 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555761 2530 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 20:34:47.562683 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555764 2530 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 20:34:47.562683 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555767 2530 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 20:34:47.562683 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555770 2530 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 20:34:47.562683 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555773 2530 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 20:34:47.562683 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555775 2530 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 20:34:47.562683 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555778 2530 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 20:34:47.562683 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:47.555781 2530 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 20:34:47.562683 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.555787 2530 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 20:34:47.562683 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.555948 2530 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 20:34:47.562683 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.558043 2530 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 20:34:47.562683 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.560615 2530 server.go:1019] "Starting client certificate rotation" Jul 10 20:34:47.563126 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.560688 2530 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 20:34:47.563126 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.560739 2530 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 20:34:47.565537 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.565520 2530 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 20:34:47.567172 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.567154 2530 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 20:34:47.581116 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.581089 2530 log.go:25] "Validated CRI v1 runtime API" Jul 10 20:34:47.583309 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.583290 2530 log.go:25] "Validated CRI v1 image API" Jul 10 20:34:47.584229 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.584213 2530 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 20:34:47.587366 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.587344 2530 fs.go:135] Filesystem UUIDs: map[5765d92b-65b2-4dc4-970a-7fca724551c4:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e48ee55f-6c7a-48fd-8bab-56ddf9949555:/dev/nvme0n1p3] Jul 10 20:34:47.587436 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.587366 2530 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 20:34:47.593243 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.593108 2530 manager.go:216] Machine: {Timestamp:2026-07-10 20:34:47.591732486 +0000 UTC m=+0.370313874 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100529 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec224e95249eaf5ef1b0ea40107573aa SystemUUID:ec224e95-249e-af5e-f1b0-ea40107573aa BootID:2986e33d-2e80-47f9-8f95-9172f575f9b2 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4b:47:79:34:31 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4b:47:79:34:31 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:9a:91:80:02:53 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 20:34:47.593243 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.593235 2530 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 20:34:47.593403 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.593387 2530 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 20:34:47.593501 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.593481 2530 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 20:34:47.596599 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.596572 2530 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 20:34:47.596787 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.596605 2530 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-64.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 20:34:47.598125 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.598109 2530 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 20:34:47.598188 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.598130 2530 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 20:34:47.598188 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.598143 2530 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 20:34:47.599069 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.599055 2530 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 20:34:47.600004 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.599990 2530 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 20:34:47.603128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.603112 2530 state_mem.go:36] "Initialized new in-memory state store" Jul 10 20:34:47.603275 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.603263 2530 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 20:34:47.603347 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.603304 2530 kubelet.go:486] "Attempting to sync node with API server" Jul 10 20:34:47.603347 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.603322 2530 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 20:34:47.603347 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.603345 2530 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 20:34:47.603483 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.603361 2530 kubelet.go:398] "Adding apiserver pod source" Jul 10 20:34:47.603483 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.603376 2530 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 20:34:47.603801 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.603786 2530 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 20:34:47.603868 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.603811 2530 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 20:34:47.606465 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.606445 2530 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 20:34:47.609340 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.609321 2530 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 20:34:47.609443 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.609364 2530 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 20:34:47.610823 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.610791 2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 20:34:47.611073 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.611060 2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 20:34:47.611116 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.611075 2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 20:34:47.611116 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.611084 2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 20:34:47.611116 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.611094 2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 20:34:47.611116 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.611101 2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 20:34:47.611116 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.611110 2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 20:34:47.611116 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.611115 2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 20:34:47.611325 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.611123 2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 20:34:47.611325 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.611130 2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 20:34:47.611325 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.611141 2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 20:34:47.611325 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.611146 2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 20:34:47.611325 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.611172 2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 20:34:47.611325 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.611178 2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 20:34:47.615069 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.615049 2530 server.go:1268] "Started kubelet" Jul 10 20:34:47.615186 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.615132 2530 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 20:34:47.615275 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.615225 2530 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 20:34:47.615326 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.615310 2530 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 20:34:47.616096 ip-10-0-139-64 systemd[1]: Started Kubernetes Kubelet. Jul 10 20:34:47.616268 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.616212 2530 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-64.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 20:34:47.616268 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:47.616191 2530 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-64.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 20:34:47.616388 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:47.616291 2530 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 20:34:47.616499 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.616480 2530 server.go:310] "Adding debug handlers to kubelet server" Jul 10 20:34:47.617092 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.617031 2530 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7d4hb" Jul 10 20:34:47.617319 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.617293 2530 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 20:34:47.622533 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:47.622516 2530 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 20:34:47.622814 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:47.621655 2530 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-64.ec2.internal.18c107f7a495a373 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-64.ec2.internal,UID:ip-10-0-139-64.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-64.ec2.internal,},FirstTimestamp:2026-07-10 20:34:47.615013747 +0000 UTC m=+0.393595135,LastTimestamp:2026-07-10 20:34:47.615013747 +0000 UTC m=+0.393595135,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-64.ec2.internal,}" Jul 10 20:34:47.623104 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.623090 2530 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 20:34:47.623104 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.623099 2530 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 20:34:47.623220 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:47.623158 2530 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-64.ec2.internal\" not found" Jul 10 20:34:47.623752 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.623736 2530 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 20:34:47.623879 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.623784 2530 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 20:34:47.623879 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.623863 2530 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 20:34:47.624028 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.624011 2530 reconstruct.go:97] "Volume reconstruction finished" Jul 10 20:34:47.624154 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.624135 2530 reconciler.go:29] "Reconciler: start to sync state" Jul 10 20:34:47.624154 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.624141 2530 factory.go:55] Registering systemd factory Jul 10 20:34:47.624309 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.624168 2530 factory.go:223] Registration of the systemd container factory successfully Jul 10 20:34:47.624414 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.624401 2530 factory.go:153] Registering CRI-O factory Jul 10 20:34:47.624453 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.624418 2530 factory.go:223] Registration of the crio container factory successfully Jul 10 20:34:47.624483 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.624466 2530 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 20:34:47.624515 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.624499 2530 factory.go:103] Registering Raw factory Jul 10 20:34:47.624548 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.624518 2530 manager.go:1196] Started watching for new ooms in manager Jul 10 20:34:47.625061 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.625043 2530 manager.go:349] Starting recovery of all containers Jul 10 20:34:47.625061 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.625062 2530 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7d4hb" Jul 10 20:34:47.633733 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:47.633684 2530 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-64.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 20:34:47.633924 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:47.633881 2530 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 20:34:47.639477 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.639094 2530 manager.go:354] Recovery completed Jul 10 20:34:47.645271 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.645252 2530 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 20:34:47.647324 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.647302 2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-64.ec2.internal" event="NodeHasSufficientMemory" Jul 10 20:34:47.647380 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.647351 2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-64.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 20:34:47.647380 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.647367 2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-64.ec2.internal" event="NodeHasSufficientPID" Jul 10 20:34:47.647953 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.647919 2530 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 20:34:47.647953 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.647938 2530 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 20:34:47.647953 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.647939 2530 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 20:34:47.648114 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.647960 2530 state_mem.go:36] "Initialized new in-memory state store" Jul 10 20:34:47.648717 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.648701 2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-64.ec2.internal" event="NodeHasSufficientMemory" Jul 10 20:34:47.648784 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.648730 2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-64.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 20:34:47.648784 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.648743 2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-64.ec2.internal" event="NodeHasSufficientPID" Jul 10 20:34:47.649455 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.649441 2530 policy_none.go:49] "None policy: Start" Jul 10 20:34:47.649937 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.649926 2530 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 20:34:47.649974 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.649945 2530 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 20:34:47.650973 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.650962 2530 policy_none.go:47] "Start" Jul 10 20:34:47.684542 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.684521 2530 manager.go:337] "Starting Device Plugin manager" Jul 10 20:34:47.684657 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:47.684570 2530 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 20:34:47.684657 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.684585 2530 server.go:87] "Starting device plugin registration server" Jul 10 20:34:47.685081 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.685066 2530 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 20:34:47.685160 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.685084 2530 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 20:34:47.685209 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.685199 2530 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 20:34:47.685319 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.685295 2530 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 20:34:47.685319 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.685310 2530 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 20:34:47.685871 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:47.685848 2530 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 20:34:47.685958 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:47.685914 2530 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-64.ec2.internal\" not found" Jul 10 20:34:47.764915 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.764877 2530 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 20:34:47.766119 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.766103 2530 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 20:34:47.766198 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.766142 2530 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 20:34:47.766198 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.766169 2530 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 20:34:47.766306 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:47.766210 2530 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 20:34:47.768464 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.768442 2530 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 20:34:47.786144 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.786112 2530 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 20:34:47.787136 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.787112 2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-64.ec2.internal" event="NodeHasSufficientMemory" Jul 10 20:34:47.787250 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.787152 2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-64.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 20:34:47.787250 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.787165 2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-64.ec2.internal" event="NodeHasSufficientPID" Jul 10 20:34:47.787250 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.787190 2530 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-64.ec2.internal" Jul 10 20:34:47.793680 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.793663 2530 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-64.ec2.internal" Jul 10 20:34:47.793763 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:47.793689 2530 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-64.ec2.internal\": node \"ip-10-0-139-64.ec2.internal\" not found" Jul 10 20:34:47.808245 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:47.808221 2530 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-64.ec2.internal\" not found" Jul 10 20:34:47.866765 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.866726 2530 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-64.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-64.ec2.internal"] Jul 10 20:34:47.866957 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.866819 2530 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 20:34:47.867879 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.867854 2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-64.ec2.internal" event="NodeHasSufficientMemory" Jul 10 20:34:47.868026 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.867892 2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-64.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 20:34:47.868026 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.867911 2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-64.ec2.internal" event="NodeHasSufficientPID" Jul 10 20:34:47.869357 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.869342 2530 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 20:34:47.869525 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.869509 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-64.ec2.internal" Jul 10 20:34:47.869592 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.869552 2530 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 20:34:47.870398 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.870383 2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-64.ec2.internal" event="NodeHasSufficientMemory" Jul 10 20:34:47.870469 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.870410 2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-64.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 20:34:47.870469 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.870429 2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-64.ec2.internal" event="NodeHasSufficientPID" Jul 10 20:34:47.870902 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.870885 2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-64.ec2.internal" event="NodeHasSufficientMemory" Jul 10 20:34:47.870902 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.870912 2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-64.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 20:34:47.871031 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.870926 2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-64.ec2.internal" event="NodeHasSufficientPID" Jul 10 20:34:47.872040 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.872027 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-64.ec2.internal" Jul 10 20:34:47.872088 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.872052 2530 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 20:34:47.872741 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.872726 2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-64.ec2.internal" event="NodeHasSufficientMemory" Jul 10 20:34:47.872814 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.872753 2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-64.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 20:34:47.872814 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.872764 2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-64.ec2.internal" event="NodeHasSufficientPID" Jul 10 20:34:47.895860 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:47.895829 2530 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-64.ec2.internal\" not found" node="ip-10-0-139-64.ec2.internal" Jul 10 20:34:47.900649 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:47.900626 2530 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-64.ec2.internal\" not found" node="ip-10-0-139-64.ec2.internal" Jul 10 20:34:47.908808 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:47.908788 2530 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-64.ec2.internal\" not found" Jul 10 20:34:47.926661 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.926586 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7b1e004b547bc6f7c2a2b3988819cb7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-64.ec2.internal\" (UID: \"7b1e004b547bc6f7c2a2b3988819cb7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-64.ec2.internal" Jul 10 20:34:47.926661 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.926619 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b1e004b547bc6f7c2a2b3988819cb7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-64.ec2.internal\" (UID: \"7b1e004b547bc6f7c2a2b3988819cb7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-64.ec2.internal" Jul 10 20:34:47.926661 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:47.926640 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9e4bdc1f22e9646521fc55e80037564b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-64.ec2.internal\" (UID: \"9e4bdc1f22e9646521fc55e80037564b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-64.ec2.internal" Jul 10 20:34:48.009051 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:48.009009 2530 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-64.ec2.internal\" not found" Jul 10 20:34:48.027651 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.027614 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7b1e004b547bc6f7c2a2b3988819cb7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-64.ec2.internal\" (UID: \"7b1e004b547bc6f7c2a2b3988819cb7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-64.ec2.internal" Jul 10 20:34:48.027820 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.027670 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b1e004b547bc6f7c2a2b3988819cb7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-64.ec2.internal\" (UID: \"7b1e004b547bc6f7c2a2b3988819cb7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-64.ec2.internal" Jul 10 20:34:48.027820 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.027717 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9e4bdc1f22e9646521fc55e80037564b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-64.ec2.internal\" (UID: \"9e4bdc1f22e9646521fc55e80037564b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-64.ec2.internal" Jul 10 20:34:48.027820 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.027722 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7b1e004b547bc6f7c2a2b3988819cb7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-64.ec2.internal\" (UID: \"7b1e004b547bc6f7c2a2b3988819cb7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-64.ec2.internal" Jul 10 20:34:48.027820 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.027742 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b1e004b547bc6f7c2a2b3988819cb7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-64.ec2.internal\" (UID: \"7b1e004b547bc6f7c2a2b3988819cb7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-64.ec2.internal" Jul 10 20:34:48.027820 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.027791 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9e4bdc1f22e9646521fc55e80037564b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-64.ec2.internal\" (UID: \"9e4bdc1f22e9646521fc55e80037564b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-64.ec2.internal" Jul 10 20:34:48.110090 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:48.110057 2530 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-64.ec2.internal\" not found" Jul 10 20:34:48.202612 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.202520 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-64.ec2.internal" Jul 10 20:34:48.203709 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.203689 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-64.ec2.internal" Jul 10 20:34:48.210659 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:48.210642 2530 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-64.ec2.internal\" not found" Jul 10 20:34:48.317353 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:48.317297 2530 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-64.ec2.internal\" not found" Jul 10 20:34:48.418430 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:48.418375 2530 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-64.ec2.internal\" not found" Jul 10 20:34:48.519039 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:48.518948 2530 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-64.ec2.internal\" not found" Jul 10 20:34:48.544709 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.544680 2530 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 20:34:48.560802 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.560770 2530 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 20:34:48.561011 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.560986 2530 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 20:34:48.561070 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.560994 2530 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 20:34:48.591680 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.591646 2530 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 20:34:48.603568 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.603536 2530 apiserver.go:52] "Watching apiserver" Jul 10 20:34:48.610720 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.610694 2530 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 20:34:48.611185 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.611163 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-gx5mn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8","openshift-cluster-node-tuning-operator/tuned-v5trs","openshift-dns/node-resolver-kpwbz","openshift-image-registry/node-ca-w9q99","openshift-multus/multus-additional-cni-plugins-47wft","openshift-network-operator/iptables-alerter-n72pf","openshift-ovn-kubernetes/ovnkube-node-n7kg6","kube-system/global-pull-secret-syncer-lmvbr","openshift-multus/multus-qmjv5","openshift-multus/network-metrics-daemon-dwvvg","openshift-network-diagnostics/network-check-target-bvktz"] Jul 10 20:34:48.612736 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.612719 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gx5mn" Jul 10 20:34:48.615512 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.614611 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" Jul 10 20:34:48.616389 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.615675 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 20:34:48.616389 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.615679 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bnxbw\"" Jul 10 20:34:48.616492 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.616372 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 20:34:48.616644 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.616626 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.617388 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.617367 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 20:34:48.617500 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.617460 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 20:34:48.617500 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.617473 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 20:34:48.617500 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.617487 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rfspx\"" Jul 10 20:34:48.618463 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.618440 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kpwbz" Jul 10 20:34:48.619223 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.619197 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 20:34:48.619223 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.619202 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4mwhm\"" Jul 10 20:34:48.619378 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.619271 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 20:34:48.619722 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.619706 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w9q99" Jul 10 20:34:48.620680 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.620664 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f57gp\"" Jul 10 20:34:48.620882 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.620866 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 20:34:48.620983 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.620920 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 20:34:48.621060 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.621039 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:48.622132 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.622113 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 20:34:48.622276 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.622132 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qqjrn\"" Jul 10 20:34:48.622276 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.622186 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 20:34:48.622276 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.622197 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 20:34:48.622479 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.622468 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n72pf" Jul 10 20:34:48.623229 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.623209 2530 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 20:34:48.623229 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.623229 2530 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-64.ec2.internal" Jul 10 20:34:48.623686 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.623668 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 20:34:48.623794 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.623669 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-26bht\"" Jul 10 20:34:48.623878 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.623832 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.624321 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.623921 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 20:34:48.624321 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.623934 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 20:34:48.624321 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.623955 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 20:34:48.624321 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.623835 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 20:34:48.624742 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.624719 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 20:34:48.624742 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.624721 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 20:34:48.624742 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.624741 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 20:34:48.624982 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.624836 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-l8kjk\"" Jul 10 20:34:48.625372 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.625357 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:34:48.625428 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:48.625411 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmvbr" podUID="7da12ef6-d8c0-4c2c-823a-33c8d885e222" Jul 10 20:34:48.626366 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.626343 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6vcb9\"" Jul 10 20:34:48.626454 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.626368 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 20:34:48.626660 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.626641 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 20:34:48.626743 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.626654 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 20:34:48.626743 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.626695 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 20:34:48.626743 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.626709 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 20:34:48.626743 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.626726 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 20:34:48.627542 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.627108 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.628731 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.628714 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:34:48.628825 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:48.628789 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvvg" podUID="93c55127-6ddc-4b15-844e-8b9585cc5d1e" Jul 10 20:34:48.629274 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.629256 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 20:34:48.629372 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.629356 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lchzf\"" Jul 10 20:34:48.630295 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.630271 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:34:48.630374 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:48.630353 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvktz" podUID="1c53c174-8a70-4166-ab02-439d93723704" Jul 10 20:34:48.632634 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.632612 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6c0b1df-8049-472c-b375-1812f1ff620b-cnibin\") pod \"multus-additional-cni-plugins-47wft\" (UID: \"d6c0b1df-8049-472c-b375-1812f1ff620b\") " pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:48.632742 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.632651 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e6b5d497-ebb0-4d0c-a589-4243df1b85ad-registration-dir\") pod \"aws-ebs-csi-driver-node-4bqn8\" (UID: \"e6b5d497-ebb0-4d0c-a589-4243df1b85ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" Jul 10 20:34:48.632742 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.632678 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e6b5d497-ebb0-4d0c-a589-4243df1b85ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-4bqn8\" (UID: \"e6b5d497-ebb0-4d0c-a589-4243df1b85ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" Jul 10 20:34:48.632742 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.632706 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfgvz\" (UniqueName: \"kubernetes.io/projected/e6b5d497-ebb0-4d0c-a589-4243df1b85ad-kube-api-access-mfgvz\") pod \"aws-ebs-csi-driver-node-4bqn8\" (UID: \"e6b5d497-ebb0-4d0c-a589-4243df1b85ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" Jul 10 20:34:48.632742 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.632735 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12401ef9-0e51-4dc0-8e83-ccb7fe160a6e-tmp-dir\") pod \"node-resolver-kpwbz\" (UID: \"12401ef9-0e51-4dc0-8e83-ccb7fe160a6e\") " pod="openshift-dns/node-resolver-kpwbz" Jul 10 20:34:48.632961 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.632759 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-etc-openvswitch\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.632961 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.632784 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-run-openvswitch\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.632961 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.632816 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-log-socket\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.632961 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.632865 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a0fc144e-a4ca-4f83-9126-9366a781f0ad-env-overrides\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.632961 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.632903 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-etc-kubernetes\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.632961 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.632942 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b2743f9-3a33-4dfa-a239-dbd0c983bd20-serviceca\") pod \"node-ca-w9q99\" (UID: \"7b2743f9-3a33-4dfa-a239-dbd0c983bd20\") " pod="openshift-image-registry/node-ca-w9q99" Jul 10 20:34:48.633242 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.632968 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e6b5d497-ebb0-4d0c-a589-4243df1b85ad-socket-dir\") pod \"aws-ebs-csi-driver-node-4bqn8\" (UID: \"e6b5d497-ebb0-4d0c-a589-4243df1b85ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" Jul 10 20:34:48.633242 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.632992 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-run-ovn\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.633242 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633040 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-etc-systemd\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.633242 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633065 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e6b5d497-ebb0-4d0c-a589-4243df1b85ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4bqn8\" (UID: \"e6b5d497-ebb0-4d0c-a589-4243df1b85ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" Jul 10 20:34:48.633242 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633090 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7da12ef6-d8c0-4c2c-823a-33c8d885e222-dbus\") pod \"global-pull-secret-syncer-lmvbr\" (UID: \"7da12ef6-d8c0-4c2c-823a-33c8d885e222\") " pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:34:48.633242 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633115 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e4d35ba8-662a-461f-b95a-7e3c54034a1f-konnectivity-ca\") pod \"konnectivity-agent-gx5mn\" (UID: \"e4d35ba8-662a-461f-b95a-7e3c54034a1f\") " pod="kube-system/konnectivity-agent-gx5mn" Jul 10 20:34:48.633242 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633139 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-host-cni-netd\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.633242 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633166 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a7e51bd2-949a-4065-860a-403fb15b8211-etc-tuned\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.633242 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633196 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fcpg\" (UniqueName: \"kubernetes.io/projected/a7e51bd2-949a-4065-860a-403fb15b8211-kube-api-access-5fcpg\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.633242 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633218 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6c0b1df-8049-472c-b375-1812f1ff620b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-47wft\" (UID: \"d6c0b1df-8049-472c-b375-1812f1ff620b\") " pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:48.633678 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633270 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfvdm\" (UniqueName: \"kubernetes.io/projected/d6c0b1df-8049-472c-b375-1812f1ff620b-kube-api-access-vfvdm\") pod \"multus-additional-cni-plugins-47wft\" (UID: \"d6c0b1df-8049-472c-b375-1812f1ff620b\") " pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:48.633678 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633302 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/df3e3f60-9d62-4173-9d10-0e2d6ebfe8dc-iptables-alerter-script\") pod \"iptables-alerter-n72pf\" (UID: \"df3e3f60-9d62-4173-9d10-0e2d6ebfe8dc\") " pod="openshift-network-operator/iptables-alerter-n72pf" Jul 10 20:34:48.633678 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633330 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-var-lib-openvswitch\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.633678 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633347 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.633678 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633364 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-host-slash\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.633678 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633379 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-etc-modprobe-d\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.633678 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633422 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6c0b1df-8049-472c-b375-1812f1ff620b-cni-binary-copy\") pod \"multus-additional-cni-plugins-47wft\" (UID: \"d6c0b1df-8049-472c-b375-1812f1ff620b\") " pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:48.633678 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633451 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-systemd-units\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.633678 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633476 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-node-log\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.633678 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633500 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-sys\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.633678 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633525 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-var-lib-kubelet\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.633678 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633556 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e6b5d497-ebb0-4d0c-a589-4243df1b85ad-sys-fs\") pod \"aws-ebs-csi-driver-node-4bqn8\" (UID: \"e6b5d497-ebb0-4d0c-a589-4243df1b85ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" Jul 10 20:34:48.633678 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633581 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a0fc144e-a4ca-4f83-9126-9366a781f0ad-ovn-node-metrics-cert\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.633678 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633621 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7da12ef6-d8c0-4c2c-823a-33c8d885e222-original-pull-secret\") pod \"global-pull-secret-syncer-lmvbr\" (UID: \"7da12ef6-d8c0-4c2c-823a-33c8d885e222\") " pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:34:48.633678 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633647 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-host-run-netns\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.633678 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633674 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-run\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.634223 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633696 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df3e3f60-9d62-4173-9d10-0e2d6ebfe8dc-host-slash\") pod \"iptables-alerter-n72pf\" (UID: \"df3e3f60-9d62-4173-9d10-0e2d6ebfe8dc\") " pod="openshift-network-operator/iptables-alerter-n72pf" Jul 10 20:34:48.634223 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633724 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-host-run-ovn-kubernetes\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.634223 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633753 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-lib-modules\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.634223 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633776 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e6b5d497-ebb0-4d0c-a589-4243df1b85ad-device-dir\") pod \"aws-ebs-csi-driver-node-4bqn8\" (UID: \"e6b5d497-ebb0-4d0c-a589-4243df1b85ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" Jul 10 20:34:48.634223 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633801 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nntzn\" (UniqueName: \"kubernetes.io/projected/a0fc144e-a4ca-4f83-9126-9366a781f0ad-kube-api-access-nntzn\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.634223 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633817 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7e51bd2-949a-4065-860a-403fb15b8211-tmp\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.634223 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.633834 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6c0b1df-8049-472c-b375-1812f1ff620b-system-cni-dir\") pod \"multus-additional-cni-plugins-47wft\" (UID: \"d6c0b1df-8049-472c-b375-1812f1ff620b\") " pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:48.634429 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.634309 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d6c0b1df-8049-472c-b375-1812f1ff620b-os-release\") pod \"multus-additional-cni-plugins-47wft\" (UID: \"d6c0b1df-8049-472c-b375-1812f1ff620b\") " pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:48.634429 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.634333 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6c0b1df-8049-472c-b375-1812f1ff620b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-47wft\" (UID: \"d6c0b1df-8049-472c-b375-1812f1ff620b\") " pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:48.634429 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.634350 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk4jx\" (UniqueName: \"kubernetes.io/projected/df3e3f60-9d62-4173-9d10-0e2d6ebfe8dc-kube-api-access-zk4jx\") pod \"iptables-alerter-n72pf\" (UID: \"df3e3f60-9d62-4173-9d10-0e2d6ebfe8dc\") " pod="openshift-network-operator/iptables-alerter-n72pf" Jul 10 20:34:48.634429 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.634369 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7da12ef6-d8c0-4c2c-823a-33c8d885e222-kubelet-config\") pod \"global-pull-secret-syncer-lmvbr\" (UID: \"7da12ef6-d8c0-4c2c-823a-33c8d885e222\") " pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:34:48.634429 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.634394 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-etc-sysctl-d\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.634600 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.634418 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8cz2\" (UniqueName: \"kubernetes.io/projected/7b2743f9-3a33-4dfa-a239-dbd0c983bd20-kube-api-access-c8cz2\") pod \"node-ca-w9q99\" (UID: \"7b2743f9-3a33-4dfa-a239-dbd0c983bd20\") " pod="openshift-image-registry/node-ca-w9q99" Jul 10 20:34:48.634600 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.634448 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e4d35ba8-662a-461f-b95a-7e3c54034a1f-agent-certs\") pod \"konnectivity-agent-gx5mn\" (UID: \"e4d35ba8-662a-461f-b95a-7e3c54034a1f\") " pod="kube-system/konnectivity-agent-gx5mn" Jul 10 20:34:48.634600 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.634468 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/12401ef9-0e51-4dc0-8e83-ccb7fe160a6e-hosts-file\") pod \"node-resolver-kpwbz\" (UID: \"12401ef9-0e51-4dc0-8e83-ccb7fe160a6e\") " pod="openshift-dns/node-resolver-kpwbz" Jul 10 20:34:48.634600 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.634490 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxfpm\" (UniqueName: \"kubernetes.io/projected/12401ef9-0e51-4dc0-8e83-ccb7fe160a6e-kube-api-access-gxfpm\") pod \"node-resolver-kpwbz\" (UID: \"12401ef9-0e51-4dc0-8e83-ccb7fe160a6e\") " pod="openshift-dns/node-resolver-kpwbz" Jul 10 20:34:48.634600 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.634524 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-host-kubelet\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.634600 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.634539 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a0fc144e-a4ca-4f83-9126-9366a781f0ad-ovnkube-script-lib\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.634600 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.634555 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-etc-sysctl-conf\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.634600 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.634575 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6c0b1df-8049-472c-b375-1812f1ff620b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-47wft\" (UID: \"d6c0b1df-8049-472c-b375-1812f1ff620b\") " pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:48.634600 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.634598 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-host-cni-bin\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.634899 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.634623 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a0fc144e-a4ca-4f83-9126-9366a781f0ad-ovnkube-config\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.634899 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.634639 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b2743f9-3a33-4dfa-a239-dbd0c983bd20-host\") pod \"node-ca-w9q99\" (UID: \"7b2743f9-3a33-4dfa-a239-dbd0c983bd20\") " pod="openshift-image-registry/node-ca-w9q99" Jul 10 20:34:48.634899 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.634661 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-run-systemd\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.634899 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.634677 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-etc-sysconfig\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.634899 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.634698 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-host\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.635172 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.635157 2530 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 20:34:48.635314 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.635294 2530 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-64.ec2.internal" Jul 10 20:34:48.635380 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.635363 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-64.ec2.internal"] Jul 10 20:34:48.635684 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.635619 2530 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 20:29:48 +0000 UTC" deadline="2028-02-14 12:32:08.164652085 +0000 UTC" Jul 10 20:34:48.635684 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.635682 2530 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14007h57m19.528972304s" Jul 10 20:34:48.637595 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.637547 2530 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 20:34:48.643787 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.643764 2530 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 20:34:48.643975 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.643959 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-64.ec2.internal"] Jul 10 20:34:48.661091 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.661065 2530 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ppk65" Jul 10 20:34:48.669639 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.669619 2530 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ppk65" Jul 10 20:34:48.724906 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.724852 2530 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 20:34:48.734943 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.734904 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-host-var-lib-kubelet\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.735107 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.734959 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-host-cni-netd\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.735107 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.734988 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a7e51bd2-949a-4065-860a-403fb15b8211-etc-tuned\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.735107 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735037 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fcpg\" (UniqueName: \"kubernetes.io/projected/a7e51bd2-949a-4065-860a-403fb15b8211-kube-api-access-5fcpg\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.735394 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735143 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6c0b1df-8049-472c-b375-1812f1ff620b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-47wft\" (UID: \"d6c0b1df-8049-472c-b375-1812f1ff620b\") " pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:48.735394 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735163 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-host-cni-netd\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.735394 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735183 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfvdm\" (UniqueName: \"kubernetes.io/projected/d6c0b1df-8049-472c-b375-1812f1ff620b-kube-api-access-vfvdm\") pod \"multus-additional-cni-plugins-47wft\" (UID: \"d6c0b1df-8049-472c-b375-1812f1ff620b\") " pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:48.735394 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735213 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/df3e3f60-9d62-4173-9d10-0e2d6ebfe8dc-iptables-alerter-script\") pod \"iptables-alerter-n72pf\" (UID: \"df3e3f60-9d62-4173-9d10-0e2d6ebfe8dc\") " pod="openshift-network-operator/iptables-alerter-n72pf" Jul 10 20:34:48.735394 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735240 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-var-lib-openvswitch\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.735394 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735268 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.735394 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735296 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh6hk\" (UniqueName: \"kubernetes.io/projected/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-kube-api-access-hh6hk\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.735394 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735324 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-host-slash\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.735394 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735350 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-etc-modprobe-d\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.735394 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735377 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6c0b1df-8049-472c-b375-1812f1ff620b-cni-binary-copy\") pod \"multus-additional-cni-plugins-47wft\" (UID: \"d6c0b1df-8049-472c-b375-1812f1ff620b\") " pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:48.735394 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735376 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.735394 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735401 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-cnibin\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.736085 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735426 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-host-run-multus-certs\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.736085 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735436 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-var-lib-openvswitch\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.736085 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735453 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-systemd-units\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.736085 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735465 2530 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 20:34:48.736085 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735479 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-node-log\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.736085 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735505 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-sys\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.736085 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735550 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-var-lib-kubelet\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.736085 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735551 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-etc-modprobe-d\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.736085 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735594 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e6b5d497-ebb0-4d0c-a589-4243df1b85ad-sys-fs\") pod \"aws-ebs-csi-driver-node-4bqn8\" (UID: \"e6b5d497-ebb0-4d0c-a589-4243df1b85ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" Jul 10 20:34:48.736085 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735611 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-systemd-units\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.736085 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735625 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-multus-socket-dir-parent\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.736085 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735657 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-node-log\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.736085 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735657 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a0fc144e-a4ca-4f83-9126-9366a781f0ad-ovn-node-metrics-cert\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.736085 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735697 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7da12ef6-d8c0-4c2c-823a-33c8d885e222-original-pull-secret\") pod \"global-pull-secret-syncer-lmvbr\" (UID: \"7da12ef6-d8c0-4c2c-823a-33c8d885e222\") " pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:34:48.736085 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735747 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-multus-daemon-config\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.736085 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735773 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsctg\" (UniqueName: \"kubernetes.io/projected/1c53c174-8a70-4166-ab02-439d93723704-kube-api-access-dsctg\") pod \"network-check-target-bvktz\" (UID: \"1c53c174-8a70-4166-ab02-439d93723704\") " pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:34:48.736085 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735803 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-host-run-netns\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.736835 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735832 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-hostroot\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.736835 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735862 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-run\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.736835 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735887 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df3e3f60-9d62-4173-9d10-0e2d6ebfe8dc-host-slash\") pod \"iptables-alerter-n72pf\" (UID: \"df3e3f60-9d62-4173-9d10-0e2d6ebfe8dc\") " pod="openshift-network-operator/iptables-alerter-n72pf" Jul 10 20:34:48.736835 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735934 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs\") pod \"network-metrics-daemon-dwvvg\" (UID: \"93c55127-6ddc-4b15-844e-8b9585cc5d1e\") " pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:34:48.736835 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735961 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dfvx\" (UniqueName: \"kubernetes.io/projected/93c55127-6ddc-4b15-844e-8b9585cc5d1e-kube-api-access-5dfvx\") pod \"network-metrics-daemon-dwvvg\" (UID: \"93c55127-6ddc-4b15-844e-8b9585cc5d1e\") " pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:34:48.736835 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735987 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-host-run-ovn-kubernetes\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.736835 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.735859 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6c0b1df-8049-472c-b375-1812f1ff620b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-47wft\" (UID: \"d6c0b1df-8049-472c-b375-1812f1ff620b\") " pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:48.736835 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.736033 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6c0b1df-8049-472c-b375-1812f1ff620b-cni-binary-copy\") pod \"multus-additional-cni-plugins-47wft\" (UID: \"d6c0b1df-8049-472c-b375-1812f1ff620b\") " pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:48.736835 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.736014 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-lib-modules\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.736835 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.736082 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-host-run-ovn-kubernetes\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.736835 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.736083 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-host-run-netns\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.736835 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.736121 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df3e3f60-9d62-4173-9d10-0e2d6ebfe8dc-host-slash\") pod \"iptables-alerter-n72pf\" (UID: \"df3e3f60-9d62-4173-9d10-0e2d6ebfe8dc\") " pod="openshift-network-operator/iptables-alerter-n72pf" Jul 10 20:34:48.736835 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.736143 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-run\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.736835 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.736147 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-sys\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.736835 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.736239 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-lib-modules\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.736835 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.736288 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-var-lib-kubelet\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.736835 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:48.736304 2530 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 20:34:48.736835 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.736338 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e6b5d497-ebb0-4d0c-a589-4243df1b85ad-sys-fs\") pod \"aws-ebs-csi-driver-node-4bqn8\" (UID: \"e6b5d497-ebb0-4d0c-a589-4243df1b85ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" Jul 10 20:34:48.738493 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.736345 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-host-slash\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.738493 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:48.736422 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7da12ef6-d8c0-4c2c-823a-33c8d885e222-original-pull-secret podName:7da12ef6-d8c0-4c2c-823a-33c8d885e222 nodeName:}" failed. No retries permitted until 2026-07-10 20:34:49.236389437 +0000 UTC m=+2.014970811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7da12ef6-d8c0-4c2c-823a-33c8d885e222-original-pull-secret") pod "global-pull-secret-syncer-lmvbr" (UID: "7da12ef6-d8c0-4c2c-823a-33c8d885e222") : object "kube-system"/"original-pull-secret" not registered Jul 10 20:34:48.738493 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.736462 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e6b5d497-ebb0-4d0c-a589-4243df1b85ad-device-dir\") pod \"aws-ebs-csi-driver-node-4bqn8\" (UID: \"e6b5d497-ebb0-4d0c-a589-4243df1b85ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" Jul 10 20:34:48.738493 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.736720 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e6b5d497-ebb0-4d0c-a589-4243df1b85ad-device-dir\") pod \"aws-ebs-csi-driver-node-4bqn8\" (UID: \"e6b5d497-ebb0-4d0c-a589-4243df1b85ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" Jul 10 20:34:48.738493 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.736784 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nntzn\" (UniqueName: \"kubernetes.io/projected/a0fc144e-a4ca-4f83-9126-9366a781f0ad-kube-api-access-nntzn\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.738493 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.736813 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7e51bd2-949a-4065-860a-403fb15b8211-tmp\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.738493 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.737146 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6c0b1df-8049-472c-b375-1812f1ff620b-system-cni-dir\") pod \"multus-additional-cni-plugins-47wft\" (UID: \"d6c0b1df-8049-472c-b375-1812f1ff620b\") " pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:48.738493 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.737184 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d6c0b1df-8049-472c-b375-1812f1ff620b-os-release\") pod \"multus-additional-cni-plugins-47wft\" (UID: \"d6c0b1df-8049-472c-b375-1812f1ff620b\") " pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:48.738493 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.737211 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6c0b1df-8049-472c-b375-1812f1ff620b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-47wft\" (UID: \"d6c0b1df-8049-472c-b375-1812f1ff620b\") " pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:48.738493 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.737249 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk4jx\" (UniqueName: \"kubernetes.io/projected/df3e3f60-9d62-4173-9d10-0e2d6ebfe8dc-kube-api-access-zk4jx\") pod \"iptables-alerter-n72pf\" (UID: \"df3e3f60-9d62-4173-9d10-0e2d6ebfe8dc\") " pod="openshift-network-operator/iptables-alerter-n72pf" Jul 10 20:34:48.738493 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.737279 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-multus-cni-dir\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.738493 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.737307 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7da12ef6-d8c0-4c2c-823a-33c8d885e222-kubelet-config\") pod \"global-pull-secret-syncer-lmvbr\" (UID: \"7da12ef6-d8c0-4c2c-823a-33c8d885e222\") " pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:34:48.738493 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.737336 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-etc-sysctl-d\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.738493 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.737363 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8cz2\" (UniqueName: \"kubernetes.io/projected/7b2743f9-3a33-4dfa-a239-dbd0c983bd20-kube-api-access-c8cz2\") pod \"node-ca-w9q99\" (UID: \"7b2743f9-3a33-4dfa-a239-dbd0c983bd20\") " pod="openshift-image-registry/node-ca-w9q99" Jul 10 20:34:48.738493 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.737389 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e4d35ba8-662a-461f-b95a-7e3c54034a1f-agent-certs\") pod \"konnectivity-agent-gx5mn\" (UID: \"e4d35ba8-662a-461f-b95a-7e3c54034a1f\") " pod="kube-system/konnectivity-agent-gx5mn" Jul 10 20:34:48.738493 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.737414 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/12401ef9-0e51-4dc0-8e83-ccb7fe160a6e-hosts-file\") pod \"node-resolver-kpwbz\" (UID: \"12401ef9-0e51-4dc0-8e83-ccb7fe160a6e\") " pod="openshift-dns/node-resolver-kpwbz" Jul 10 20:34:48.739511 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.737438 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxfpm\" (UniqueName: \"kubernetes.io/projected/12401ef9-0e51-4dc0-8e83-ccb7fe160a6e-kube-api-access-gxfpm\") pod \"node-resolver-kpwbz\" (UID: \"12401ef9-0e51-4dc0-8e83-ccb7fe160a6e\") " pod="openshift-dns/node-resolver-kpwbz" Jul 10 20:34:48.739511 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.736922 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/df3e3f60-9d62-4173-9d10-0e2d6ebfe8dc-iptables-alerter-script\") pod \"iptables-alerter-n72pf\" (UID: \"df3e3f60-9d62-4173-9d10-0e2d6ebfe8dc\") " pod="openshift-network-operator/iptables-alerter-n72pf" Jul 10 20:34:48.739511 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.737527 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d6c0b1df-8049-472c-b375-1812f1ff620b-os-release\") pod \"multus-additional-cni-plugins-47wft\" (UID: \"d6c0b1df-8049-472c-b375-1812f1ff620b\") " pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:48.739511 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.737563 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-host-run-k8s-cni-cncf-io\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.739511 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.737899 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7da12ef6-d8c0-4c2c-823a-33c8d885e222-kubelet-config\") pod \"global-pull-secret-syncer-lmvbr\" (UID: \"7da12ef6-d8c0-4c2c-823a-33c8d885e222\") " pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:34:48.739511 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.737906 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/12401ef9-0e51-4dc0-8e83-ccb7fe160a6e-hosts-file\") pod \"node-resolver-kpwbz\" (UID: \"12401ef9-0e51-4dc0-8e83-ccb7fe160a6e\") " pod="openshift-dns/node-resolver-kpwbz" Jul 10 20:34:48.739511 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.737939 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-host-var-lib-cni-multus\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.739511 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.737971 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-host-kubelet\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.739511 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.737999 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a0fc144e-a4ca-4f83-9126-9366a781f0ad-ovnkube-script-lib\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.739511 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.738026 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-etc-sysctl-d\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.739511 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.738055 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-etc-sysctl-conf\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.739511 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.738073 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6c0b1df-8049-472c-b375-1812f1ff620b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-47wft\" (UID: \"d6c0b1df-8049-472c-b375-1812f1ff620b\") " pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:48.739511 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.738086 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6c0b1df-8049-472c-b375-1812f1ff620b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-47wft\" (UID: \"d6c0b1df-8049-472c-b375-1812f1ff620b\") " pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:48.739511 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.738111 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-multus-conf-dir\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.739511 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.738142 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-host-kubelet\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.739511 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.738191 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-host-cni-bin\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.739511 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.738225 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-host-cni-bin\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.740811 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.738219 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a0fc144e-a4ca-4f83-9126-9366a781f0ad-ovnkube-config\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.740811 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.738298 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b2743f9-3a33-4dfa-a239-dbd0c983bd20-host\") pod \"node-ca-w9q99\" (UID: \"7b2743f9-3a33-4dfa-a239-dbd0c983bd20\") " pod="openshift-image-registry/node-ca-w9q99" Jul 10 20:34:48.740811 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.738332 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-host-run-netns\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.740811 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.738361 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-run-systemd\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.740811 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.738456 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-etc-sysconfig\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.740811 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.738492 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-host\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.740811 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.738518 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6c0b1df-8049-472c-b375-1812f1ff620b-cnibin\") pod \"multus-additional-cni-plugins-47wft\" (UID: \"d6c0b1df-8049-472c-b375-1812f1ff620b\") " pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:48.740811 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.738570 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e6b5d497-ebb0-4d0c-a589-4243df1b85ad-registration-dir\") pod \"aws-ebs-csi-driver-node-4bqn8\" (UID: \"e6b5d497-ebb0-4d0c-a589-4243df1b85ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" Jul 10 20:34:48.740811 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.738604 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e6b5d497-ebb0-4d0c-a589-4243df1b85ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-4bqn8\" (UID: \"e6b5d497-ebb0-4d0c-a589-4243df1b85ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" Jul 10 20:34:48.740811 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.738636 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfgvz\" (UniqueName: \"kubernetes.io/projected/e6b5d497-ebb0-4d0c-a589-4243df1b85ad-kube-api-access-mfgvz\") pod \"aws-ebs-csi-driver-node-4bqn8\" (UID: \"e6b5d497-ebb0-4d0c-a589-4243df1b85ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" Jul 10 20:34:48.740811 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.738670 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12401ef9-0e51-4dc0-8e83-ccb7fe160a6e-tmp-dir\") pod \"node-resolver-kpwbz\" (UID: \"12401ef9-0e51-4dc0-8e83-ccb7fe160a6e\") " pod="openshift-dns/node-resolver-kpwbz" Jul 10 20:34:48.740811 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.738685 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b2743f9-3a33-4dfa-a239-dbd0c983bd20-host\") pod \"node-ca-w9q99\" (UID: \"7b2743f9-3a33-4dfa-a239-dbd0c983bd20\") " pod="openshift-image-registry/node-ca-w9q99" Jul 10 20:34:48.740811 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.738737 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-etc-openvswitch\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.740811 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.738756 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-run-systemd\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.740811 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.738636 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a0fc144e-a4ca-4f83-9126-9366a781f0ad-ovnkube-script-lib\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.740811 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.738826 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6c0b1df-8049-472c-b375-1812f1ff620b-system-cni-dir\") pod \"multus-additional-cni-plugins-47wft\" (UID: \"d6c0b1df-8049-472c-b375-1812f1ff620b\") " pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:48.740811 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739122 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-etc-openvswitch\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.741343 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739134 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12401ef9-0e51-4dc0-8e83-ccb7fe160a6e-tmp-dir\") pod \"node-resolver-kpwbz\" (UID: \"12401ef9-0e51-4dc0-8e83-ccb7fe160a6e\") " pod="openshift-dns/node-resolver-kpwbz" Jul 10 20:34:48.741343 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739211 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-host\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.741343 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739217 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e6b5d497-ebb0-4d0c-a589-4243df1b85ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-4bqn8\" (UID: \"e6b5d497-ebb0-4d0c-a589-4243df1b85ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" Jul 10 20:34:48.741343 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739259 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-run-openvswitch\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.741343 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739269 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-etc-sysconfig\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.741343 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739292 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-log-socket\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.741343 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739335 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a0fc144e-a4ca-4f83-9126-9366a781f0ad-env-overrides\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.741343 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739340 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-log-socket\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.741343 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739351 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-run-openvswitch\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.741343 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739398 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-etc-kubernetes\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.741343 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739411 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6c0b1df-8049-472c-b375-1812f1ff620b-cnibin\") pod \"multus-additional-cni-plugins-47wft\" (UID: \"d6c0b1df-8049-472c-b375-1812f1ff620b\") " pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:48.741343 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739433 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b2743f9-3a33-4dfa-a239-dbd0c983bd20-serviceca\") pod \"node-ca-w9q99\" (UID: \"7b2743f9-3a33-4dfa-a239-dbd0c983bd20\") " pod="openshift-image-registry/node-ca-w9q99" Jul 10 20:34:48.741343 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739505 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e6b5d497-ebb0-4d0c-a589-4243df1b85ad-socket-dir\") pod \"aws-ebs-csi-driver-node-4bqn8\" (UID: \"e6b5d497-ebb0-4d0c-a589-4243df1b85ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" Jul 10 20:34:48.741343 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739540 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-system-cni-dir\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.741343 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739567 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-etc-sysctl-conf\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.741343 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739573 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-run-ovn\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.741343 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739587 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a7e51bd2-949a-4065-860a-403fb15b8211-etc-tuned\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.741343 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739603 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-etc-systemd\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.742028 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739643 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-etc-kubernetes\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.742028 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739682 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e6b5d497-ebb0-4d0c-a589-4243df1b85ad-registration-dir\") pod \"aws-ebs-csi-driver-node-4bqn8\" (UID: \"e6b5d497-ebb0-4d0c-a589-4243df1b85ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" Jul 10 20:34:48.742028 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739695 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7e51bd2-949a-4065-860a-403fb15b8211-tmp\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.742028 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739693 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e6b5d497-ebb0-4d0c-a589-4243df1b85ad-socket-dir\") pod \"aws-ebs-csi-driver-node-4bqn8\" (UID: \"e6b5d497-ebb0-4d0c-a589-4243df1b85ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" Jul 10 20:34:48.742028 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739738 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e6b5d497-ebb0-4d0c-a589-4243df1b85ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4bqn8\" (UID: \"e6b5d497-ebb0-4d0c-a589-4243df1b85ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" Jul 10 20:34:48.742028 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739757 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a0fc144e-a4ca-4f83-9126-9366a781f0ad-ovnkube-config\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.742028 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739770 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-etc-kubernetes\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.742028 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739759 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7e51bd2-949a-4065-860a-403fb15b8211-etc-systemd\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.742028 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739826 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7da12ef6-d8c0-4c2c-823a-33c8d885e222-dbus\") pod \"global-pull-secret-syncer-lmvbr\" (UID: \"7da12ef6-d8c0-4c2c-823a-33c8d885e222\") " pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:34:48.742028 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739855 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e4d35ba8-662a-461f-b95a-7e3c54034a1f-konnectivity-ca\") pod \"konnectivity-agent-gx5mn\" (UID: \"e4d35ba8-662a-461f-b95a-7e3c54034a1f\") " pod="kube-system/konnectivity-agent-gx5mn" Jul 10 20:34:48.742028 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739881 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a0fc144e-a4ca-4f83-9126-9366a781f0ad-env-overrides\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.742028 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739892 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6c0b1df-8049-472c-b375-1812f1ff620b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-47wft\" (UID: \"d6c0b1df-8049-472c-b375-1812f1ff620b\") " pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:48.742028 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739884 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-os-release\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.742028 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739958 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a0fc144e-a4ca-4f83-9126-9366a781f0ad-run-ovn\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.742028 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.739967 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e6b5d497-ebb0-4d0c-a589-4243df1b85ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4bqn8\" (UID: \"e6b5d497-ebb0-4d0c-a589-4243df1b85ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" Jul 10 20:34:48.742028 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.740002 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-cni-binary-copy\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.742028 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.740031 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-host-var-lib-cni-bin\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.742652 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.740097 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b2743f9-3a33-4dfa-a239-dbd0c983bd20-serviceca\") pod \"node-ca-w9q99\" (UID: \"7b2743f9-3a33-4dfa-a239-dbd0c983bd20\") " pod="openshift-image-registry/node-ca-w9q99" Jul 10 20:34:48.742652 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.740136 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7da12ef6-d8c0-4c2c-823a-33c8d885e222-dbus\") pod \"global-pull-secret-syncer-lmvbr\" (UID: \"7da12ef6-d8c0-4c2c-823a-33c8d885e222\") " pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:34:48.742652 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.740238 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a0fc144e-a4ca-4f83-9126-9366a781f0ad-ovn-node-metrics-cert\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.742652 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.740526 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e4d35ba8-662a-461f-b95a-7e3c54034a1f-konnectivity-ca\") pod \"konnectivity-agent-gx5mn\" (UID: \"e4d35ba8-662a-461f-b95a-7e3c54034a1f\") " pod="kube-system/konnectivity-agent-gx5mn" Jul 10 20:34:48.742652 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.741809 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e4d35ba8-662a-461f-b95a-7e3c54034a1f-agent-certs\") pod \"konnectivity-agent-gx5mn\" (UID: \"e4d35ba8-662a-461f-b95a-7e3c54034a1f\") " pod="kube-system/konnectivity-agent-gx5mn" Jul 10 20:34:48.744693 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.744301 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fcpg\" (UniqueName: \"kubernetes.io/projected/a7e51bd2-949a-4065-860a-403fb15b8211-kube-api-access-5fcpg\") pod \"tuned-v5trs\" (UID: \"a7e51bd2-949a-4065-860a-403fb15b8211\") " pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.745196 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.745158 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfvdm\" (UniqueName: \"kubernetes.io/projected/d6c0b1df-8049-472c-b375-1812f1ff620b-kube-api-access-vfvdm\") pod \"multus-additional-cni-plugins-47wft\" (UID: \"d6c0b1df-8049-472c-b375-1812f1ff620b\") " pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:48.746327 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.745958 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nntzn\" (UniqueName: \"kubernetes.io/projected/a0fc144e-a4ca-4f83-9126-9366a781f0ad-kube-api-access-nntzn\") pod \"ovnkube-node-n7kg6\" (UID: \"a0fc144e-a4ca-4f83-9126-9366a781f0ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.746562 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.746541 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8cz2\" (UniqueName: \"kubernetes.io/projected/7b2743f9-3a33-4dfa-a239-dbd0c983bd20-kube-api-access-c8cz2\") pod \"node-ca-w9q99\" (UID: \"7b2743f9-3a33-4dfa-a239-dbd0c983bd20\") " pod="openshift-image-registry/node-ca-w9q99" Jul 10 20:34:48.746656 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.746597 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk4jx\" (UniqueName: \"kubernetes.io/projected/df3e3f60-9d62-4173-9d10-0e2d6ebfe8dc-kube-api-access-zk4jx\") pod \"iptables-alerter-n72pf\" (UID: \"df3e3f60-9d62-4173-9d10-0e2d6ebfe8dc\") " pod="openshift-network-operator/iptables-alerter-n72pf" Jul 10 20:34:48.747210 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.747186 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxfpm\" (UniqueName: \"kubernetes.io/projected/12401ef9-0e51-4dc0-8e83-ccb7fe160a6e-kube-api-access-gxfpm\") pod \"node-resolver-kpwbz\" (UID: \"12401ef9-0e51-4dc0-8e83-ccb7fe160a6e\") " pod="openshift-dns/node-resolver-kpwbz" Jul 10 20:34:48.747303 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.747237 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfgvz\" (UniqueName: \"kubernetes.io/projected/e6b5d497-ebb0-4d0c-a589-4243df1b85ad-kube-api-access-mfgvz\") pod \"aws-ebs-csi-driver-node-4bqn8\" (UID: \"e6b5d497-ebb0-4d0c-a589-4243df1b85ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" Jul 10 20:34:48.760045 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.760022 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:34:48.834972 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:48.834921 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1e004b547bc6f7c2a2b3988819cb7d.slice/crio-a1eb4053d99bdb601ea1ed70a4f043f148f098d4edc09d35db20dfbaebb96e10 WatchSource:0}: Error finding container a1eb4053d99bdb601ea1ed70a4f043f148f098d4edc09d35db20dfbaebb96e10: Status 404 returned error can't find the container with id a1eb4053d99bdb601ea1ed70a4f043f148f098d4edc09d35db20dfbaebb96e10 Jul 10 20:34:48.835201 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:48.835179 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4bdc1f22e9646521fc55e80037564b.slice/crio-4be85734ba57188360d4e350c0f77b87c04b31422e1403a89e8a737ae12a5b85 WatchSource:0}: Error finding container 4be85734ba57188360d4e350c0f77b87c04b31422e1403a89e8a737ae12a5b85: Status 404 returned error can't find the container with id 4be85734ba57188360d4e350c0f77b87c04b31422e1403a89e8a737ae12a5b85 Jul 10 20:34:48.840477 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.840446 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-system-cni-dir\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.840603 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.840485 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-etc-kubernetes\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.840603 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.840514 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-os-release\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.840603 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.840540 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-cni-binary-copy\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.840603 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.840566 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-host-var-lib-cni-bin\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.840603 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.840593 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-host-var-lib-kubelet\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.840856 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.840607 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-etc-kubernetes\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.840856 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.840639 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh6hk\" (UniqueName: \"kubernetes.io/projected/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-kube-api-access-hh6hk\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.840856 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.840651 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-host-var-lib-cni-bin\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.840856 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.840632 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-os-release\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.840856 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.840691 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-host-var-lib-kubelet\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.840856 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.840704 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-system-cni-dir\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.840856 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.840777 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-cnibin\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.840856 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.840808 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-host-run-multus-certs\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.841577 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.840835 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-multus-socket-dir-parent\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.841577 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.840937 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-cnibin\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.841577 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.840978 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-host-run-multus-certs\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.841577 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.841000 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-multus-daemon-config\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.841577 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.841023 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-multus-socket-dir-parent\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.841577 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.841032 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsctg\" (UniqueName: \"kubernetes.io/projected/1c53c174-8a70-4166-ab02-439d93723704-kube-api-access-dsctg\") pod \"network-check-target-bvktz\" (UID: \"1c53c174-8a70-4166-ab02-439d93723704\") " pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:34:48.841577 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.841061 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-hostroot\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.841577 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.841110 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs\") pod \"network-metrics-daemon-dwvvg\" (UID: \"93c55127-6ddc-4b15-844e-8b9585cc5d1e\") " pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:34:48.841577 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.841137 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dfvx\" (UniqueName: \"kubernetes.io/projected/93c55127-6ddc-4b15-844e-8b9585cc5d1e-kube-api-access-5dfvx\") pod \"network-metrics-daemon-dwvvg\" (UID: \"93c55127-6ddc-4b15-844e-8b9585cc5d1e\") " pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:34:48.841577 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.841157 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-hostroot\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.841577 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.841219 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-multus-cni-dir\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.841577 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:48.841248 2530 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:34:48.841577 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.841249 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-cni-binary-copy\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.841577 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:48.841316 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs podName:93c55127-6ddc-4b15-844e-8b9585cc5d1e nodeName:}" failed. No retries permitted until 2026-07-10 20:34:49.341296721 +0000 UTC m=+2.119878111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs") pod "network-metrics-daemon-dwvvg" (UID: "93c55127-6ddc-4b15-844e-8b9585cc5d1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:34:48.841577 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.841356 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-multus-cni-dir\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.841577 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.841261 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-host-run-k8s-cni-cncf-io\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.841577 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.841451 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-host-var-lib-cni-multus\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.841577 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.841510 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-multus-conf-dir\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.842523 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.841550 2530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 20:34:48.842523 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.841563 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-host-run-netns\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.842523 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.841563 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-host-run-k8s-cni-cncf-io\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.842523 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.841563 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-multus-daemon-config\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.842523 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.841605 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-multus-conf-dir\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.842523 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.841667 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-host-run-netns\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.842523 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.841685 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-host-var-lib-cni-multus\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.849908 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:48.849877 2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 20:34:48.849908 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:48.849911 2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 20:34:48.850044 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:48.849922 2530 projected.go:196] Error preparing data for projected volume kube-api-access-dsctg for pod openshift-network-diagnostics/network-check-target-bvktz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:34:48.850044 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:48.849974 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c53c174-8a70-4166-ab02-439d93723704-kube-api-access-dsctg podName:1c53c174-8a70-4166-ab02-439d93723704 nodeName:}" failed. No retries permitted until 2026-07-10 20:34:49.34996098 +0000 UTC m=+2.128542357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dsctg" (UniqueName: "kubernetes.io/projected/1c53c174-8a70-4166-ab02-439d93723704-kube-api-access-dsctg") pod "network-check-target-bvktz" (UID: "1c53c174-8a70-4166-ab02-439d93723704") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:34:48.851965 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.851945 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh6hk\" (UniqueName: \"kubernetes.io/projected/9b7a55bf-4524-49b1-9147-2e0bdb51d2db-kube-api-access-hh6hk\") pod \"multus-qmjv5\" (UID: \"9b7a55bf-4524-49b1-9147-2e0bdb51d2db\") " pod="openshift-multus/multus-qmjv5" Jul 10 20:34:48.852208 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.852192 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dfvx\" (UniqueName: \"kubernetes.io/projected/93c55127-6ddc-4b15-844e-8b9585cc5d1e-kube-api-access-5dfvx\") pod \"network-metrics-daemon-dwvvg\" (UID: \"93c55127-6ddc-4b15-844e-8b9585cc5d1e\") " pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:34:48.931774 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.931727 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gx5mn" Jul 10 20:34:48.940343 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:48.940314 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d35ba8_662a_461f_b95a_7e3c54034a1f.slice/crio-0ae7c9a35dcbb2516ce4c5d3bc7eb5e9c3d3feea8cee334c940ac8f95b9cde68 WatchSource:0}: Error finding container 0ae7c9a35dcbb2516ce4c5d3bc7eb5e9c3d3feea8cee334c940ac8f95b9cde68: Status 404 returned error can't find the container with id 0ae7c9a35dcbb2516ce4c5d3bc7eb5e9c3d3feea8cee334c940ac8f95b9cde68 Jul 10 20:34:48.956604 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.956578 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" Jul 10 20:34:48.965420 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:48.965390 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b5d497_ebb0_4d0c_a589_4243df1b85ad.slice/crio-967f725acb7472679fbb9c97119f8fa3e434b3a657dfcc0674d49d45ade9c5a0 WatchSource:0}: Error finding container 967f725acb7472679fbb9c97119f8fa3e434b3a657dfcc0674d49d45ade9c5a0: Status 404 returned error can't find the container with id 967f725acb7472679fbb9c97119f8fa3e434b3a657dfcc0674d49d45ade9c5a0 Jul 10 20:34:48.967198 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.967177 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v5trs" Jul 10 20:34:48.974206 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:48.974182 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e51bd2_949a_4065_860a_403fb15b8211.slice/crio-3a69553689e026f50c85122351fc847fc02887a6e7ea013a9bfa248aec60c183 WatchSource:0}: Error finding container 3a69553689e026f50c85122351fc847fc02887a6e7ea013a9bfa248aec60c183: Status 404 returned error can't find the container with id 3a69553689e026f50c85122351fc847fc02887a6e7ea013a9bfa248aec60c183 Jul 10 20:34:48.980460 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:48.980439 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kpwbz" Jul 10 20:34:48.989346 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:48.989318 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12401ef9_0e51_4dc0_8e83_ccb7fe160a6e.slice/crio-55bbc010d9e6ed68c5610e220d8627fb437fa4c891218ab665a38504b31cd1eb WatchSource:0}: Error finding container 55bbc010d9e6ed68c5610e220d8627fb437fa4c891218ab665a38504b31cd1eb: Status 404 returned error can't find the container with id 55bbc010d9e6ed68c5610e220d8627fb437fa4c891218ab665a38504b31cd1eb Jul 10 20:34:49.004222 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:49.004181 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w9q99" Jul 10 20:34:49.012424 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:49.012400 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2743f9_3a33_4dfa_a239_dbd0c983bd20.slice/crio-9cf4cbe26fe768ac3eb6726ecf6b14e1afcb23eed191b464a06c205cd0b80e96 WatchSource:0}: Error finding container 9cf4cbe26fe768ac3eb6726ecf6b14e1afcb23eed191b464a06c205cd0b80e96: Status 404 returned error can't find the container with id 9cf4cbe26fe768ac3eb6726ecf6b14e1afcb23eed191b464a06c205cd0b80e96 Jul 10 20:34:49.013413 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:49.013398 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-47wft" Jul 10 20:34:49.021164 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:49.021131 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c0b1df_8049_472c_b375_1812f1ff620b.slice/crio-2ded25dd9ffdb7ee491340b2d783b202f0c8975a406882c8098649332348ac87 WatchSource:0}: Error finding container 2ded25dd9ffdb7ee491340b2d783b202f0c8975a406882c8098649332348ac87: Status 404 returned error can't find the container with id 2ded25dd9ffdb7ee491340b2d783b202f0c8975a406882c8098649332348ac87 Jul 10 20:34:49.032446 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:49.032425 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n72pf" Jul 10 20:34:49.035132 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:49.035111 2530 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 20:34:49.040476 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:49.040453 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf3e3f60_9d62_4173_9d10_0e2d6ebfe8dc.slice/crio-727ccf1fdd13ae33e41d4d1c132e0c305c7902a86b32e50673662f85f8502740 WatchSource:0}: Error finding container 727ccf1fdd13ae33e41d4d1c132e0c305c7902a86b32e50673662f85f8502740: Status 404 returned error can't find the container with id 727ccf1fdd13ae33e41d4d1c132e0c305c7902a86b32e50673662f85f8502740 Jul 10 20:34:49.066533 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:49.066498 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qmjv5" Jul 10 20:34:49.075319 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:49.075264 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7a55bf_4524_49b1_9147_2e0bdb51d2db.slice/crio-16dd170861e9ea2668331f58c8b8a2310b5485e098fe759fd3d92a52a3f692a5 WatchSource:0}: Error finding container 16dd170861e9ea2668331f58c8b8a2310b5485e098fe759fd3d92a52a3f692a5: Status 404 returned error can't find the container with id 16dd170861e9ea2668331f58c8b8a2310b5485e098fe759fd3d92a52a3f692a5 Jul 10 20:34:49.118382 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:34:49.118350 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0fc144e_a4ca_4f83_9126_9366a781f0ad.slice/crio-21b6e7c89769f082248a668ff3ec0a69293049b172aa632e1481a2756d3e14af WatchSource:0}: Error finding container 21b6e7c89769f082248a668ff3ec0a69293049b172aa632e1481a2756d3e14af: Status 404 returned error can't find the container with id 21b6e7c89769f082248a668ff3ec0a69293049b172aa632e1481a2756d3e14af Jul 10 20:34:49.243468 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:49.243430 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7da12ef6-d8c0-4c2c-823a-33c8d885e222-original-pull-secret\") pod \"global-pull-secret-syncer-lmvbr\" (UID: \"7da12ef6-d8c0-4c2c-823a-33c8d885e222\") " pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:34:49.243652 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:49.243540 2530 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 20:34:49.243652 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:49.243609 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7da12ef6-d8c0-4c2c-823a-33c8d885e222-original-pull-secret podName:7da12ef6-d8c0-4c2c-823a-33c8d885e222 nodeName:}" failed. No retries permitted until 2026-07-10 20:34:50.2435953 +0000 UTC m=+3.022176673 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7da12ef6-d8c0-4c2c-823a-33c8d885e222-original-pull-secret") pod "global-pull-secret-syncer-lmvbr" (UID: "7da12ef6-d8c0-4c2c-823a-33c8d885e222") : object "kube-system"/"original-pull-secret" not registered Jul 10 20:34:49.344279 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:49.344192 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs\") pod \"network-metrics-daemon-dwvvg\" (UID: \"93c55127-6ddc-4b15-844e-8b9585cc5d1e\") " pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:34:49.344957 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:49.344591 2530 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:34:49.344957 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:49.344697 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs podName:93c55127-6ddc-4b15-844e-8b9585cc5d1e nodeName:}" failed. No retries permitted until 2026-07-10 20:34:50.344673958 +0000 UTC m=+3.123255347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs") pod "network-metrics-daemon-dwvvg" (UID: "93c55127-6ddc-4b15-844e-8b9585cc5d1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:34:49.431721 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:49.431550 2530 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 20:34:49.445587 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:49.445550 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsctg\" (UniqueName: \"kubernetes.io/projected/1c53c174-8a70-4166-ab02-439d93723704-kube-api-access-dsctg\") pod \"network-check-target-bvktz\" (UID: \"1c53c174-8a70-4166-ab02-439d93723704\") " pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:34:49.445796 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:49.445777 2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 20:34:49.445865 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:49.445805 2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 20:34:49.445865 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:49.445819 2530 projected.go:196] Error preparing data for projected volume kube-api-access-dsctg for pod openshift-network-diagnostics/network-check-target-bvktz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:34:49.445985 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:49.445898 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c53c174-8a70-4166-ab02-439d93723704-kube-api-access-dsctg podName:1c53c174-8a70-4166-ab02-439d93723704 nodeName:}" failed. No retries permitted until 2026-07-10 20:34:50.445859132 +0000 UTC m=+3.224440524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dsctg" (UniqueName: "kubernetes.io/projected/1c53c174-8a70-4166-ab02-439d93723704-kube-api-access-dsctg") pod "network-check-target-bvktz" (UID: "1c53c174-8a70-4166-ab02-439d93723704") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:34:49.673243 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:49.673125 2530 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 20:29:49 +0000 UTC" deadline="2028-01-30 21:44:54.152027765 +0000 UTC" Jul 10 20:34:49.673243 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:49.673166 2530 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13657h10m4.478866655s" Jul 10 20:34:49.777157 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:49.776590 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:34:49.777157 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:49.776742 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvvg" podUID="93c55127-6ddc-4b15-844e-8b9585cc5d1e" Jul 10 20:34:49.787626 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:49.787558 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n72pf" event={"ID":"df3e3f60-9d62-4173-9d10-0e2d6ebfe8dc","Type":"ContainerStarted","Data":"727ccf1fdd13ae33e41d4d1c132e0c305c7902a86b32e50673662f85f8502740"} Jul 10 20:34:49.803509 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:49.803464 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47wft" event={"ID":"d6c0b1df-8049-472c-b375-1812f1ff620b","Type":"ContainerStarted","Data":"2ded25dd9ffdb7ee491340b2d783b202f0c8975a406882c8098649332348ac87"} Jul 10 20:34:49.806684 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:49.806587 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v5trs" event={"ID":"a7e51bd2-949a-4065-860a-403fb15b8211","Type":"ContainerStarted","Data":"3a69553689e026f50c85122351fc847fc02887a6e7ea013a9bfa248aec60c183"} Jul 10 20:34:49.814587 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:49.814496 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" event={"ID":"e6b5d497-ebb0-4d0c-a589-4243df1b85ad","Type":"ContainerStarted","Data":"967f725acb7472679fbb9c97119f8fa3e434b3a657dfcc0674d49d45ade9c5a0"} Jul 10 20:34:49.824741 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:49.824693 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-64.ec2.internal" event={"ID":"7b1e004b547bc6f7c2a2b3988819cb7d","Type":"ContainerStarted","Data":"a1eb4053d99bdb601ea1ed70a4f043f148f098d4edc09d35db20dfbaebb96e10"} Jul 10 20:34:49.840901 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:49.840858 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-64.ec2.internal" event={"ID":"9e4bdc1f22e9646521fc55e80037564b","Type":"ContainerStarted","Data":"4be85734ba57188360d4e350c0f77b87c04b31422e1403a89e8a737ae12a5b85"} Jul 10 20:34:49.853152 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:49.853026 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" event={"ID":"a0fc144e-a4ca-4f83-9126-9366a781f0ad","Type":"ContainerStarted","Data":"21b6e7c89769f082248a668ff3ec0a69293049b172aa632e1481a2756d3e14af"} Jul 10 20:34:49.869690 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:49.868033 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qmjv5" event={"ID":"9b7a55bf-4524-49b1-9147-2e0bdb51d2db","Type":"ContainerStarted","Data":"16dd170861e9ea2668331f58c8b8a2310b5485e098fe759fd3d92a52a3f692a5"} Jul 10 20:34:49.885967 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:49.885925 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w9q99" event={"ID":"7b2743f9-3a33-4dfa-a239-dbd0c983bd20","Type":"ContainerStarted","Data":"9cf4cbe26fe768ac3eb6726ecf6b14e1afcb23eed191b464a06c205cd0b80e96"} Jul 10 20:34:49.904368 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:49.904290 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kpwbz" event={"ID":"12401ef9-0e51-4dc0-8e83-ccb7fe160a6e","Type":"ContainerStarted","Data":"55bbc010d9e6ed68c5610e220d8627fb437fa4c891218ab665a38504b31cd1eb"} Jul 10 20:34:49.914945 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:49.911095 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gx5mn" event={"ID":"e4d35ba8-662a-461f-b95a-7e3c54034a1f","Type":"ContainerStarted","Data":"0ae7c9a35dcbb2516ce4c5d3bc7eb5e9c3d3feea8cee334c940ac8f95b9cde68"} Jul 10 20:34:50.253379 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:50.253222 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7da12ef6-d8c0-4c2c-823a-33c8d885e222-original-pull-secret\") pod \"global-pull-secret-syncer-lmvbr\" (UID: \"7da12ef6-d8c0-4c2c-823a-33c8d885e222\") " pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:34:50.253593 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:50.253405 2530 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 20:34:50.253593 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:50.253490 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7da12ef6-d8c0-4c2c-823a-33c8d885e222-original-pull-secret podName:7da12ef6-d8c0-4c2c-823a-33c8d885e222 nodeName:}" failed. No retries permitted until 2026-07-10 20:34:52.253467681 +0000 UTC m=+5.032049064 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7da12ef6-d8c0-4c2c-823a-33c8d885e222-original-pull-secret") pod "global-pull-secret-syncer-lmvbr" (UID: "7da12ef6-d8c0-4c2c-823a-33c8d885e222") : object "kube-system"/"original-pull-secret" not registered Jul 10 20:34:50.354977 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:50.354644 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs\") pod \"network-metrics-daemon-dwvvg\" (UID: \"93c55127-6ddc-4b15-844e-8b9585cc5d1e\") " pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:34:50.354977 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:50.354866 2530 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:34:50.354977 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:50.354962 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs podName:93c55127-6ddc-4b15-844e-8b9585cc5d1e nodeName:}" failed. No retries permitted until 2026-07-10 20:34:52.3549402 +0000 UTC m=+5.133521577 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs") pod "network-metrics-daemon-dwvvg" (UID: "93c55127-6ddc-4b15-844e-8b9585cc5d1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:34:50.455604 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:50.455565 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsctg\" (UniqueName: \"kubernetes.io/projected/1c53c174-8a70-4166-ab02-439d93723704-kube-api-access-dsctg\") pod \"network-check-target-bvktz\" (UID: \"1c53c174-8a70-4166-ab02-439d93723704\") " pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:34:50.455819 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:50.455793 2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 20:34:50.455819 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:50.455814 2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 20:34:50.455966 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:50.455827 2530 projected.go:196] Error preparing data for projected volume kube-api-access-dsctg for pod openshift-network-diagnostics/network-check-target-bvktz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:34:50.455966 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:50.455910 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c53c174-8a70-4166-ab02-439d93723704-kube-api-access-dsctg podName:1c53c174-8a70-4166-ab02-439d93723704 nodeName:}" failed. No retries permitted until 2026-07-10 20:34:52.455871854 +0000 UTC m=+5.234453255 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dsctg" (UniqueName: "kubernetes.io/projected/1c53c174-8a70-4166-ab02-439d93723704-kube-api-access-dsctg") pod "network-check-target-bvktz" (UID: "1c53c174-8a70-4166-ab02-439d93723704") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:34:50.674043 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:50.673918 2530 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 20:29:49 +0000 UTC" deadline="2027-12-18 07:56:36.489644299 +0000 UTC" Jul 10 20:34:50.674043 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:50.673989 2530 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12611h21m45.815660226s" Jul 10 20:34:50.720232 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:50.720196 2530 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 20:34:50.766592 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:50.766519 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:34:50.766802 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:50.766715 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmvbr" podUID="7da12ef6-d8c0-4c2c-823a-33c8d885e222" Jul 10 20:34:50.767298 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:50.767224 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:34:50.767396 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:50.767332 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvktz" podUID="1c53c174-8a70-4166-ab02-439d93723704" Jul 10 20:34:51.769009 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:51.768400 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:34:51.769009 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:51.768558 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvvg" podUID="93c55127-6ddc-4b15-844e-8b9585cc5d1e" Jul 10 20:34:52.272550 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:52.272502 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7da12ef6-d8c0-4c2c-823a-33c8d885e222-original-pull-secret\") pod \"global-pull-secret-syncer-lmvbr\" (UID: \"7da12ef6-d8c0-4c2c-823a-33c8d885e222\") " pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:34:52.272748 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:52.272711 2530 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 20:34:52.272883 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:52.272778 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7da12ef6-d8c0-4c2c-823a-33c8d885e222-original-pull-secret podName:7da12ef6-d8c0-4c2c-823a-33c8d885e222 nodeName:}" failed. No retries permitted until 2026-07-10 20:34:56.272760137 +0000 UTC m=+9.051341513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7da12ef6-d8c0-4c2c-823a-33c8d885e222-original-pull-secret") pod "global-pull-secret-syncer-lmvbr" (UID: "7da12ef6-d8c0-4c2c-823a-33c8d885e222") : object "kube-system"/"original-pull-secret" not registered Jul 10 20:34:52.373588 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:52.373544 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs\") pod \"network-metrics-daemon-dwvvg\" (UID: \"93c55127-6ddc-4b15-844e-8b9585cc5d1e\") " pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:34:52.373784 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:52.373728 2530 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:34:52.373842 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:52.373799 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs podName:93c55127-6ddc-4b15-844e-8b9585cc5d1e nodeName:}" failed. No retries permitted until 2026-07-10 20:34:56.373779827 +0000 UTC m=+9.152361216 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs") pod "network-metrics-daemon-dwvvg" (UID: "93c55127-6ddc-4b15-844e-8b9585cc5d1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:34:52.474046 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:52.474006 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsctg\" (UniqueName: \"kubernetes.io/projected/1c53c174-8a70-4166-ab02-439d93723704-kube-api-access-dsctg\") pod \"network-check-target-bvktz\" (UID: \"1c53c174-8a70-4166-ab02-439d93723704\") " pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:34:52.474248 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:52.474203 2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 20:34:52.474248 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:52.474228 2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 20:34:52.474248 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:52.474242 2530 projected.go:196] Error preparing data for projected volume kube-api-access-dsctg for pod openshift-network-diagnostics/network-check-target-bvktz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:34:52.474402 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:52.474306 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c53c174-8a70-4166-ab02-439d93723704-kube-api-access-dsctg podName:1c53c174-8a70-4166-ab02-439d93723704 nodeName:}" failed. No retries permitted until 2026-07-10 20:34:56.474285943 +0000 UTC m=+9.252867318 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dsctg" (UniqueName: "kubernetes.io/projected/1c53c174-8a70-4166-ab02-439d93723704-kube-api-access-dsctg") pod "network-check-target-bvktz" (UID: "1c53c174-8a70-4166-ab02-439d93723704") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:34:52.767416 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:52.767328 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:34:52.767416 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:52.767362 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:34:52.767656 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:52.767481 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvktz" podUID="1c53c174-8a70-4166-ab02-439d93723704" Jul 10 20:34:52.767656 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:52.767589 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmvbr" podUID="7da12ef6-d8c0-4c2c-823a-33c8d885e222" Jul 10 20:34:53.767312 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:53.766717 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:34:53.767312 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:53.766897 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvvg" podUID="93c55127-6ddc-4b15-844e-8b9585cc5d1e" Jul 10 20:34:54.767648 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:54.767122 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:34:54.767648 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:54.767280 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmvbr" podUID="7da12ef6-d8c0-4c2c-823a-33c8d885e222" Jul 10 20:34:54.767648 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:54.767301 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:34:54.767648 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:54.767488 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvktz" podUID="1c53c174-8a70-4166-ab02-439d93723704" Jul 10 20:34:55.767555 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:55.766925 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:34:55.767555 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:55.767088 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvvg" podUID="93c55127-6ddc-4b15-844e-8b9585cc5d1e" Jul 10 20:34:56.315481 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:56.315345 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7da12ef6-d8c0-4c2c-823a-33c8d885e222-original-pull-secret\") pod \"global-pull-secret-syncer-lmvbr\" (UID: \"7da12ef6-d8c0-4c2c-823a-33c8d885e222\") " pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:34:56.315694 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:56.315607 2530 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 20:34:56.315761 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:56.315704 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7da12ef6-d8c0-4c2c-823a-33c8d885e222-original-pull-secret podName:7da12ef6-d8c0-4c2c-823a-33c8d885e222 nodeName:}" failed. No retries permitted until 2026-07-10 20:35:04.315682553 +0000 UTC m=+17.094263941 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7da12ef6-d8c0-4c2c-823a-33c8d885e222-original-pull-secret") pod "global-pull-secret-syncer-lmvbr" (UID: "7da12ef6-d8c0-4c2c-823a-33c8d885e222") : object "kube-system"/"original-pull-secret" not registered Jul 10 20:34:56.416312 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:56.416272 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs\") pod \"network-metrics-daemon-dwvvg\" (UID: \"93c55127-6ddc-4b15-844e-8b9585cc5d1e\") " pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:34:56.416521 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:56.416480 2530 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:34:56.416602 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:56.416563 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs podName:93c55127-6ddc-4b15-844e-8b9585cc5d1e nodeName:}" failed. No retries permitted until 2026-07-10 20:35:04.416540985 +0000 UTC m=+17.195122362 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs") pod "network-metrics-daemon-dwvvg" (UID: "93c55127-6ddc-4b15-844e-8b9585cc5d1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:34:56.517297 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:56.517252 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsctg\" (UniqueName: \"kubernetes.io/projected/1c53c174-8a70-4166-ab02-439d93723704-kube-api-access-dsctg\") pod \"network-check-target-bvktz\" (UID: \"1c53c174-8a70-4166-ab02-439d93723704\") " pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:34:56.517490 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:56.517455 2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 20:34:56.517490 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:56.517479 2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 20:34:56.517624 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:56.517495 2530 projected.go:196] Error preparing data for projected volume kube-api-access-dsctg for pod openshift-network-diagnostics/network-check-target-bvktz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:34:56.517624 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:56.517558 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c53c174-8a70-4166-ab02-439d93723704-kube-api-access-dsctg podName:1c53c174-8a70-4166-ab02-439d93723704 nodeName:}" failed. No retries permitted until 2026-07-10 20:35:04.517539291 +0000 UTC m=+17.296120686 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dsctg" (UniqueName: "kubernetes.io/projected/1c53c174-8a70-4166-ab02-439d93723704-kube-api-access-dsctg") pod "network-check-target-bvktz" (UID: "1c53c174-8a70-4166-ab02-439d93723704") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:34:56.767208 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:56.767123 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:34:56.767373 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:56.767260 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvktz" podUID="1c53c174-8a70-4166-ab02-439d93723704" Jul 10 20:34:56.767373 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:56.767360 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:34:56.767511 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:56.767476 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmvbr" podUID="7da12ef6-d8c0-4c2c-823a-33c8d885e222" Jul 10 20:34:57.767870 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:57.767815 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:34:57.768343 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:57.767959 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvvg" podUID="93c55127-6ddc-4b15-844e-8b9585cc5d1e" Jul 10 20:34:58.766958 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:58.766920 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:34:58.767343 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:58.766946 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:34:58.768145 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:58.767948 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvktz" podUID="1c53c174-8a70-4166-ab02-439d93723704" Jul 10 20:34:58.768145 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:58.768113 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmvbr" podUID="7da12ef6-d8c0-4c2c-823a-33c8d885e222" Jul 10 20:34:59.766577 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:34:59.766544 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:34:59.766748 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:34:59.766688 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvvg" podUID="93c55127-6ddc-4b15-844e-8b9585cc5d1e" Jul 10 20:35:00.766975 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:00.766945 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:35:00.767339 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:00.766945 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:35:00.767339 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:00.767052 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmvbr" podUID="7da12ef6-d8c0-4c2c-823a-33c8d885e222" Jul 10 20:35:00.767339 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:00.767133 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvktz" podUID="1c53c174-8a70-4166-ab02-439d93723704" Jul 10 20:35:01.766499 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:01.766462 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:35:01.766695 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:01.766592 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvvg" podUID="93c55127-6ddc-4b15-844e-8b9585cc5d1e" Jul 10 20:35:02.767025 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:02.766795 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:35:02.767466 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:02.766827 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:35:02.767466 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:02.767130 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvktz" podUID="1c53c174-8a70-4166-ab02-439d93723704" Jul 10 20:35:02.767466 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:02.767243 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmvbr" podUID="7da12ef6-d8c0-4c2c-823a-33c8d885e222" Jul 10 20:35:03.766706 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:03.766670 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:35:03.766919 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:03.766819 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvvg" podUID="93c55127-6ddc-4b15-844e-8b9585cc5d1e" Jul 10 20:35:04.378893 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:04.378851 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7da12ef6-d8c0-4c2c-823a-33c8d885e222-original-pull-secret\") pod \"global-pull-secret-syncer-lmvbr\" (UID: \"7da12ef6-d8c0-4c2c-823a-33c8d885e222\") " pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:35:04.379374 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:04.379021 2530 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 20:35:04.379374 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:04.379112 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7da12ef6-d8c0-4c2c-823a-33c8d885e222-original-pull-secret podName:7da12ef6-d8c0-4c2c-823a-33c8d885e222 nodeName:}" failed. No retries permitted until 2026-07-10 20:35:20.379089019 +0000 UTC m=+33.157670411 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7da12ef6-d8c0-4c2c-823a-33c8d885e222-original-pull-secret") pod "global-pull-secret-syncer-lmvbr" (UID: "7da12ef6-d8c0-4c2c-823a-33c8d885e222") : object "kube-system"/"original-pull-secret" not registered Jul 10 20:35:04.479230 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:04.479183 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs\") pod \"network-metrics-daemon-dwvvg\" (UID: \"93c55127-6ddc-4b15-844e-8b9585cc5d1e\") " pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:35:04.479413 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:04.479379 2530 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:35:04.479472 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:04.479460 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs podName:93c55127-6ddc-4b15-844e-8b9585cc5d1e nodeName:}" failed. No retries permitted until 2026-07-10 20:35:20.479438878 +0000 UTC m=+33.258020265 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs") pod "network-metrics-daemon-dwvvg" (UID: "93c55127-6ddc-4b15-844e-8b9585cc5d1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:35:04.579551 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:04.579512 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsctg\" (UniqueName: \"kubernetes.io/projected/1c53c174-8a70-4166-ab02-439d93723704-kube-api-access-dsctg\") pod \"network-check-target-bvktz\" (UID: \"1c53c174-8a70-4166-ab02-439d93723704\") " pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:35:04.579767 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:04.579675 2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 20:35:04.579767 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:04.579694 2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 20:35:04.579767 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:04.579705 2530 projected.go:196] Error preparing data for projected volume kube-api-access-dsctg for pod openshift-network-diagnostics/network-check-target-bvktz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:35:04.579767 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:04.579761 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c53c174-8a70-4166-ab02-439d93723704-kube-api-access-dsctg podName:1c53c174-8a70-4166-ab02-439d93723704 nodeName:}" failed. No retries permitted until 2026-07-10 20:35:20.579743514 +0000 UTC m=+33.358324890 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dsctg" (UniqueName: "kubernetes.io/projected/1c53c174-8a70-4166-ab02-439d93723704-kube-api-access-dsctg") pod "network-check-target-bvktz" (UID: "1c53c174-8a70-4166-ab02-439d93723704") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:35:04.767227 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:04.767144 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:35:04.767401 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:04.767144 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:35:04.767401 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:04.767285 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmvbr" podUID="7da12ef6-d8c0-4c2c-823a-33c8d885e222" Jul 10 20:35:04.767401 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:04.767365 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvktz" podUID="1c53c174-8a70-4166-ab02-439d93723704" Jul 10 20:35:05.766669 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:05.766633 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:35:05.767150 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:05.766784 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvvg" podUID="93c55127-6ddc-4b15-844e-8b9585cc5d1e" Jul 10 20:35:06.766720 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:06.766680 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:35:06.766720 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:06.766715 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:35:06.767204 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:06.766799 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmvbr" podUID="7da12ef6-d8c0-4c2c-823a-33c8d885e222" Jul 10 20:35:06.767204 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:06.766938 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvktz" podUID="1c53c174-8a70-4166-ab02-439d93723704" Jul 10 20:35:07.768227 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:07.768190 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:35:07.768708 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:07.768323 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvvg" podUID="93c55127-6ddc-4b15-844e-8b9585cc5d1e" Jul 10 20:35:08.766954 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:08.766913 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:35:08.767152 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:08.766913 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:35:08.767152 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:08.767057 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvktz" podUID="1c53c174-8a70-4166-ab02-439d93723704" Jul 10 20:35:08.767261 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:08.767147 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmvbr" podUID="7da12ef6-d8c0-4c2c-823a-33c8d885e222" Jul 10 20:35:09.767379 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:09.767352 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:35:09.767822 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:09.767491 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvvg" podUID="93c55127-6ddc-4b15-844e-8b9585cc5d1e" Jul 10 20:35:09.968081 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:09.968029 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v5trs" event={"ID":"a7e51bd2-949a-4065-860a-403fb15b8211","Type":"ContainerStarted","Data":"070f37fa6a6ed747db1bbfb3f3593ed18bb461ef00887e8b8a46ff705c429a00"} Jul 10 20:35:09.970712 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:09.970682 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-64.ec2.internal" event={"ID":"9e4bdc1f22e9646521fc55e80037564b","Type":"ContainerStarted","Data":"7fdd5a96dd484951ac444f2d80bb3b1ad6ae2b2fb2111fcd30ae4a51932d641b"} Jul 10 20:35:09.972148 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:09.972114 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qmjv5" event={"ID":"9b7a55bf-4524-49b1-9147-2e0bdb51d2db","Type":"ContainerStarted","Data":"5076c2255bf0580329a1991e9d1675e8369c48f8d8bd24bf85162f4a7e99cf5b"} Jul 10 20:35:09.984950 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:09.984866 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-v5trs" podStartSLOduration=1.4683047550000001 podStartE2EDuration="21.984852206s" totalImagesPullingTime="20.516547451s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:35:09.984650286 +0000 UTC m=+22.763231681" watchObservedRunningTime="2026-07-10 20:35:09.984852206 +0000 UTC m=+22.763433600" Jul 10 20:35:10.003225 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:10.002047 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qmjv5" podStartSLOduration=1.555217037 podStartE2EDuration="22.002024879s" totalImagesPullingTime="20.446807842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:35:10.000507555 +0000 UTC m=+22.779088970" watchObservedRunningTime="2026-07-10 20:35:10.002024879 +0000 UTC m=+22.780606274" Jul 10 20:35:10.013743 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:10.013691 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-64.ec2.internal" podStartSLOduration=21.013674929 podStartE2EDuration="21.013674929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:35:10.013188453 +0000 UTC m=+22.791769872" watchObservedRunningTime="2026-07-10 20:35:10.013674929 +0000 UTC m=+22.792256322" Jul 10 20:35:10.767122 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:10.766819 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:35:10.767247 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:10.766884 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:35:10.767247 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:10.767225 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmvbr" podUID="7da12ef6-d8c0-4c2c-823a-33c8d885e222" Jul 10 20:35:10.767355 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:10.767265 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvktz" podUID="1c53c174-8a70-4166-ab02-439d93723704" Jul 10 20:35:10.978621 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:10.978590 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n7kg6_a0fc144e-a4ca-4f83-9126-9366a781f0ad/ovn-acl-logging/0.log" Jul 10 20:35:10.979412 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:10.979008 2530 generic.go:356] "Generic (PLEG): container finished" podID="a0fc144e-a4ca-4f83-9126-9366a781f0ad" containerID="a21ab381a88b62649e211178754ba5a1319cb2b9fe29c66a45c504fa923dfe17" exitCode=1 Jul 10 20:35:10.979412 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:10.979105 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" event={"ID":"a0fc144e-a4ca-4f83-9126-9366a781f0ad","Type":"ContainerStarted","Data":"4ff0759b16fb96201c086513058c977108b7e55be3e84ee91ed1d4e83c372812"} Jul 10 20:35:10.979412 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:10.979180 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" event={"ID":"a0fc144e-a4ca-4f83-9126-9366a781f0ad","Type":"ContainerStarted","Data":"26659f030b34b9a9e1eac31a8c0f3dd9fac0ddfaddb3a055136c2f8c6274afb2"} Jul 10 20:35:10.979412 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:10.979229 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" event={"ID":"a0fc144e-a4ca-4f83-9126-9366a781f0ad","Type":"ContainerStarted","Data":"4eb7aedd44e8f6b16edb078846d547cf55ad23efe2d31a1682ed56ec5205e8ca"} Jul 10 20:35:10.979412 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:10.979274 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" event={"ID":"a0fc144e-a4ca-4f83-9126-9366a781f0ad","Type":"ContainerStarted","Data":"e37834e5873bd4db14f449bfa77b75b8bc8ed5408aac9ec661e82d262b71b6a0"} Jul 10 20:35:10.979412 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:10.979320 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" event={"ID":"a0fc144e-a4ca-4f83-9126-9366a781f0ad","Type":"ContainerDied","Data":"a21ab381a88b62649e211178754ba5a1319cb2b9fe29c66a45c504fa923dfe17"} Jul 10 20:35:10.979412 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:10.979364 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" event={"ID":"a0fc144e-a4ca-4f83-9126-9366a781f0ad","Type":"ContainerStarted","Data":"df102e2c0dadfe3fdc2696b71131ba1676c2f522030915369ea8955e5f0a0b19"} Jul 10 20:35:10.980772 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:10.980713 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w9q99" event={"ID":"7b2743f9-3a33-4dfa-a239-dbd0c983bd20","Type":"ContainerStarted","Data":"d3f1d819c570a93cbeb9a601e3a417208fc9b18b31dedb9222f284b1183d2b26"} Jul 10 20:35:10.982965 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:10.982574 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kpwbz" event={"ID":"12401ef9-0e51-4dc0-8e83-ccb7fe160a6e","Type":"ContainerStarted","Data":"a9d6b579be56fa44dd462fb2996c7958e655d40ef589ab213fd48bdb4e61b7eb"} Jul 10 20:35:10.984438 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:10.984416 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gx5mn" event={"ID":"e4d35ba8-662a-461f-b95a-7e3c54034a1f","Type":"ContainerStarted","Data":"21be38efc2ba09dfaf43f037721a22a7cdc11ff34a313fadd76620433d34b42a"} Jul 10 20:35:10.986090 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:10.986067 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n72pf" event={"ID":"df3e3f60-9d62-4173-9d10-0e2d6ebfe8dc","Type":"ContainerStarted","Data":"84b9eda80f8e8910bb968460250b7e476c1153f6b1d491c23c69c908de252218"} Jul 10 20:35:10.987919 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:10.987897 2530 generic.go:356] "Generic (PLEG): container finished" podID="d6c0b1df-8049-472c-b375-1812f1ff620b" containerID="0fbbb9c69d9d753d1caa879907b29e56afdb016d48e670eed611fe34a52090a6" exitCode=0 Jul 10 20:35:10.988031 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:10.987983 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47wft" event={"ID":"d6c0b1df-8049-472c-b375-1812f1ff620b","Type":"ContainerDied","Data":"0fbbb9c69d9d753d1caa879907b29e56afdb016d48e670eed611fe34a52090a6"} Jul 10 20:35:10.990386 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:10.990356 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" event={"ID":"e6b5d497-ebb0-4d0c-a589-4243df1b85ad","Type":"ContainerStarted","Data":"4979adbac7f46c9f165953fe9f987f104c73b5945f8dd7f5ecd35f7e3108f3ee"} Jul 10 20:35:10.991951 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:10.991927 2530 generic.go:356] "Generic (PLEG): container finished" podID="7b1e004b547bc6f7c2a2b3988819cb7d" containerID="f05a4266ea92988989d9c9d55d702072573037dbf1b06ecf22c824b4075ece2e" exitCode=0 Jul 10 20:35:10.992044 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:10.991973 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-64.ec2.internal" event={"ID":"7b1e004b547bc6f7c2a2b3988819cb7d","Type":"ContainerDied","Data":"f05a4266ea92988989d9c9d55d702072573037dbf1b06ecf22c824b4075ece2e"} Jul 10 20:35:10.995066 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:10.995022 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-w9q99" podStartSLOduration=2.554858531 podStartE2EDuration="22.995008654s" totalImagesPullingTime="20.440150123s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:35:10.994052353 +0000 UTC m=+23.772633748" watchObservedRunningTime="2026-07-10 20:35:10.995008654 +0000 UTC m=+23.773590098" Jul 10 20:35:11.021537 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:11.021477 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gx5mn" podStartSLOduration=2.473313573 podStartE2EDuration="23.02145731s" totalImagesPullingTime="20.548143737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:35:11.008001047 +0000 UTC m=+23.786582443" watchObservedRunningTime="2026-07-10 20:35:11.02145731 +0000 UTC m=+23.800038705" Jul 10 20:35:11.033479 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:11.033425 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kpwbz" podStartSLOduration=2.534297592 podStartE2EDuration="23.033408899s" totalImagesPullingTime="20.499111307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:35:11.033164499 +0000 UTC m=+23.811745893" watchObservedRunningTime="2026-07-10 20:35:11.033408899 +0000 UTC m=+23.811990293" Jul 10 20:35:11.064561 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:11.064516 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-n72pf" podStartSLOduration=2.61651527 podStartE2EDuration="23.064499305s" totalImagesPullingTime="20.447984035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:35:11.064116608 +0000 UTC m=+23.842698001" watchObservedRunningTime="2026-07-10 20:35:11.064499305 +0000 UTC m=+23.843080699" Jul 10 20:35:11.200523 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:11.200482 2530 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gx5mn_e4d35ba8-662a-461f-b95a-7e3c54034a1f/konnectivity-agent/0.log" limit=1024 Jul 10 20:35:11.282428 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:11.282326 2530 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gx5mn_e4d35ba8-662a-461f-b95a-7e3c54034a1f/konnectivity-agent/0.log" limit=1024 Jul 10 20:35:11.333393 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:11.333352 2530 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gx5mn_e4d35ba8-662a-461f-b95a-7e3c54034a1f/konnectivity-agent/0.log" limit=1024 Jul 10 20:35:11.656762 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:11.656734 2530 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 20:35:11.695409 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:11.695365 2530 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gx5mn_e4d35ba8-662a-461f-b95a-7e3c54034a1f/konnectivity-agent/0.log" limit=1024 Jul 10 20:35:11.700276 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:11.700173 2530 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T20:35:11.656755372Z","UUID":"6e85b3a1-4bc3-4748-b040-241dc9990488","Handler":null,"Name":"","Endpoint":""} Jul 10 20:35:11.701781 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:11.701763 2530 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 20:35:11.701925 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:11.701799 2530 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 20:35:11.767379 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:11.767344 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:35:11.767560 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:11.767500 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvvg" podUID="93c55127-6ddc-4b15-844e-8b9585cc5d1e" Jul 10 20:35:11.796870 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:11.796755 2530 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gx5mn_e4d35ba8-662a-461f-b95a-7e3c54034a1f/konnectivity-agent/0.log" limit=1024 Jul 10 20:35:11.996814 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:11.996771 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" event={"ID":"e6b5d497-ebb0-4d0c-a589-4243df1b85ad","Type":"ContainerStarted","Data":"041d53912c6e31f985952c03e7f9412de655a2066d0b86e2fae54ae50ab64dab"} Jul 10 20:35:11.998974 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:11.998914 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-64.ec2.internal" event={"ID":"7b1e004b547bc6f7c2a2b3988819cb7d","Type":"ContainerStarted","Data":"a00cb3bfb9e651bacf884c0ff771096e434caa01aa8490be763104647081fb3f"} Jul 10 20:35:12.014585 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:12.014530 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-64.ec2.internal" podStartSLOduration=23.014513287 podStartE2EDuration="23.014513287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:35:12.014171427 +0000 UTC m=+24.792752815" watchObservedRunningTime="2026-07-10 20:35:12.014513287 +0000 UTC m=+24.793094680" Jul 10 20:35:12.766995 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:12.766957 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:35:12.767295 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:12.767068 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmvbr" podUID="7da12ef6-d8c0-4c2c-823a-33c8d885e222" Jul 10 20:35:12.767295 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:12.767129 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:35:12.767393 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:12.767284 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvktz" podUID="1c53c174-8a70-4166-ab02-439d93723704" Jul 10 20:35:13.206314 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:13.206239 2530 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gx5mn_e4d35ba8-662a-461f-b95a-7e3c54034a1f/konnectivity-agent/0.log" limit=1024 Jul 10 20:35:13.304200 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:13.304162 2530 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gx5mn_e4d35ba8-662a-461f-b95a-7e3c54034a1f/konnectivity-agent/0.log" limit=1024 Jul 10 20:35:13.767222 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:13.767192 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:35:13.767522 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:13.767337 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvvg" podUID="93c55127-6ddc-4b15-844e-8b9585cc5d1e" Jul 10 20:35:14.006885 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:14.006858 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n7kg6_a0fc144e-a4ca-4f83-9126-9366a781f0ad/ovn-acl-logging/0.log" Jul 10 20:35:14.007318 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:14.007284 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" event={"ID":"a0fc144e-a4ca-4f83-9126-9366a781f0ad","Type":"ContainerStarted","Data":"45765291fdaa9b80f0bde4ce40240005295fcafe5dcb43e1e5efdf51d617d35b"} Jul 10 20:35:14.009573 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:14.009547 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" event={"ID":"e6b5d497-ebb0-4d0c-a589-4243df1b85ad","Type":"ContainerStarted","Data":"1201c20f7938701c9e95f253a27149b5dae5d48d5443b9acc566e2099a8452be"} Jul 10 20:35:14.026722 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:14.026623 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4bqn8" podStartSLOduration=3.060120339 podStartE2EDuration="26.026607557s" totalImagesPullingTime="22.966487218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:34:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:35:14.025555662 +0000 UTC m=+26.804137055" watchObservedRunningTime="2026-07-10 20:35:14.026607557 +0000 UTC m=+26.805188951" Jul 10 20:35:14.105261 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:14.105218 2530 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gx5mn" Jul 10 20:35:14.105996 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:14.105966 2530 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gx5mn" Jul 10 20:35:14.660136 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:14.660102 2530 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gx5mn_e4d35ba8-662a-461f-b95a-7e3c54034a1f/konnectivity-agent/0.log" limit=1024 Jul 10 20:35:14.759205 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:14.759171 2530 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gx5mn_e4d35ba8-662a-461f-b95a-7e3c54034a1f/konnectivity-agent/0.log" limit=1024 Jul 10 20:35:14.767305 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:14.767260 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:35:14.767305 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:14.767282 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:35:14.767533 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:14.767382 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmvbr" podUID="7da12ef6-d8c0-4c2c-823a-33c8d885e222" Jul 10 20:35:14.767533 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:14.767506 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvktz" podUID="1c53c174-8a70-4166-ab02-439d93723704" Jul 10 20:35:15.011863 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:15.011754 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gx5mn" Jul 10 20:35:15.012287 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:15.012261 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gx5mn" Jul 10 20:35:15.766783 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:15.766563 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:35:15.767472 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:15.766817 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvvg" podUID="93c55127-6ddc-4b15-844e-8b9585cc5d1e" Jul 10 20:35:16.017176 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:16.017105 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n7kg6_a0fc144e-a4ca-4f83-9126-9366a781f0ad/ovn-acl-logging/0.log" Jul 10 20:35:16.017543 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:16.017505 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" event={"ID":"a0fc144e-a4ca-4f83-9126-9366a781f0ad","Type":"ContainerStarted","Data":"7ad27f8b85faa7d4c1886ad0653b6144f14b5a9ba91074114940c4f6d0e6a631"} Jul 10 20:35:16.017734 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:16.017711 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:35:16.017894 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:16.017774 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:35:16.017972 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:16.017960 2530 scope.go:117] "RemoveContainer" containerID="a21ab381a88b62649e211178754ba5a1319cb2b9fe29c66a45c504fa923dfe17" Jul 10 20:35:16.019469 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:16.019447 2530 generic.go:356] "Generic (PLEG): container finished" podID="d6c0b1df-8049-472c-b375-1812f1ff620b" containerID="a3c48fd3f81732087b4fee31413c92491e117b52b5afec488a30411a84ae1564" exitCode=0 Jul 10 20:35:16.019577 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:16.019527 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47wft" event={"ID":"d6c0b1df-8049-472c-b375-1812f1ff620b","Type":"ContainerDied","Data":"a3c48fd3f81732087b4fee31413c92491e117b52b5afec488a30411a84ae1564"} Jul 10 20:35:16.037005 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:16.036982 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:35:16.767281 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:16.767107 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:35:16.767753 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:16.767108 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:35:16.767753 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:16.767365 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvktz" podUID="1c53c174-8a70-4166-ab02-439d93723704" Jul 10 20:35:16.767753 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:16.767433 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmvbr" podUID="7da12ef6-d8c0-4c2c-823a-33c8d885e222" Jul 10 20:35:17.028667 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:17.028598 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n7kg6_a0fc144e-a4ca-4f83-9126-9366a781f0ad/ovn-acl-logging/0.log" Jul 10 20:35:17.029035 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:17.029008 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" event={"ID":"a0fc144e-a4ca-4f83-9126-9366a781f0ad","Type":"ContainerStarted","Data":"ca1845fd3af9ef29ff6db6b38406858685844e8ce8d15f40803dd289e0963824"} Jul 10 20:35:17.029303 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:17.029283 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:35:17.044985 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:17.044957 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:35:17.056666 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:17.056610 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" podStartSLOduration=8.116941619 podStartE2EDuration="29.0565943s" totalImagesPullingTime="20.939652681s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:35:17.056508291 +0000 UTC m=+29.835089698" watchObservedRunningTime="2026-07-10 20:35:17.0565943 +0000 UTC m=+29.835175685" Jul 10 20:35:17.324602 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:17.324565 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lmvbr"] Jul 10 20:35:17.324779 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:17.324713 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:35:17.324828 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:17.324798 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmvbr" podUID="7da12ef6-d8c0-4c2c-823a-33c8d885e222" Jul 10 20:35:17.327263 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:17.327239 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bvktz"] Jul 10 20:35:17.327388 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:17.327329 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:35:17.327436 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:17.327401 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvktz" podUID="1c53c174-8a70-4166-ab02-439d93723704" Jul 10 20:35:17.335756 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:17.335732 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dwvvg"] Jul 10 20:35:17.335907 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:17.335854 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:35:17.335975 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:17.335951 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvvg" podUID="93c55127-6ddc-4b15-844e-8b9585cc5d1e" Jul 10 20:35:18.035148 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:18.035113 2530 generic.go:356] "Generic (PLEG): container finished" podID="d6c0b1df-8049-472c-b375-1812f1ff620b" containerID="bb62cf9656fe0ea03c5ea47d1947f6efb0dee3137b73b42448dfed7018d4aae6" exitCode=0 Jul 10 20:35:18.035686 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:18.035193 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47wft" event={"ID":"d6c0b1df-8049-472c-b375-1812f1ff620b","Type":"ContainerDied","Data":"bb62cf9656fe0ea03c5ea47d1947f6efb0dee3137b73b42448dfed7018d4aae6"} Jul 10 20:35:18.767148 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:18.767110 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:35:18.767148 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:18.767154 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:35:18.767418 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:18.767107 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:35:18.767418 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:18.767246 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvvg" podUID="93c55127-6ddc-4b15-844e-8b9585cc5d1e" Jul 10 20:35:18.767418 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:18.767325 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmvbr" podUID="7da12ef6-d8c0-4c2c-823a-33c8d885e222" Jul 10 20:35:18.767418 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:18.767388 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvktz" podUID="1c53c174-8a70-4166-ab02-439d93723704" Jul 10 20:35:20.042021 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:20.041991 2530 generic.go:356] "Generic (PLEG): container finished" podID="d6c0b1df-8049-472c-b375-1812f1ff620b" containerID="08b6aeb467d9a9fe151b2d8b5a68f2137c79df38306663a52cc90a9dd3f7e91e" exitCode=0 Jul 10 20:35:20.042413 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:20.042055 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47wft" event={"ID":"d6c0b1df-8049-472c-b375-1812f1ff620b","Type":"ContainerDied","Data":"08b6aeb467d9a9fe151b2d8b5a68f2137c79df38306663a52cc90a9dd3f7e91e"} Jul 10 20:35:20.394482 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:20.394382 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7da12ef6-d8c0-4c2c-823a-33c8d885e222-original-pull-secret\") pod \"global-pull-secret-syncer-lmvbr\" (UID: \"7da12ef6-d8c0-4c2c-823a-33c8d885e222\") " pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:35:20.394630 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:20.394541 2530 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 20:35:20.394630 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:20.394613 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7da12ef6-d8c0-4c2c-823a-33c8d885e222-original-pull-secret podName:7da12ef6-d8c0-4c2c-823a-33c8d885e222 nodeName:}" failed. No retries permitted until 2026-07-10 20:35:52.394597308 +0000 UTC m=+65.173178679 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7da12ef6-d8c0-4c2c-823a-33c8d885e222-original-pull-secret") pod "global-pull-secret-syncer-lmvbr" (UID: "7da12ef6-d8c0-4c2c-823a-33c8d885e222") : object "kube-system"/"original-pull-secret" not registered Jul 10 20:35:20.495384 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:20.495343 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs\") pod \"network-metrics-daemon-dwvvg\" (UID: \"93c55127-6ddc-4b15-844e-8b9585cc5d1e\") " pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:35:20.495554 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:20.495454 2530 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:35:20.495554 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:20.495506 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs podName:93c55127-6ddc-4b15-844e-8b9585cc5d1e nodeName:}" failed. No retries permitted until 2026-07-10 20:35:52.495492166 +0000 UTC m=+65.274073538 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs") pod "network-metrics-daemon-dwvvg" (UID: "93c55127-6ddc-4b15-844e-8b9585cc5d1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:35:20.596164 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:20.596122 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsctg\" (UniqueName: \"kubernetes.io/projected/1c53c174-8a70-4166-ab02-439d93723704-kube-api-access-dsctg\") pod \"network-check-target-bvktz\" (UID: \"1c53c174-8a70-4166-ab02-439d93723704\") " pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:35:20.596341 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:20.596296 2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 20:35:20.596341 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:20.596319 2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 20:35:20.596341 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:20.596328 2530 projected.go:196] Error preparing data for projected volume kube-api-access-dsctg for pod openshift-network-diagnostics/network-check-target-bvktz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:35:20.596444 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:20.596384 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c53c174-8a70-4166-ab02-439d93723704-kube-api-access-dsctg podName:1c53c174-8a70-4166-ab02-439d93723704 nodeName:}" failed. No retries permitted until 2026-07-10 20:35:52.596370852 +0000 UTC m=+65.374952223 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dsctg" (UniqueName: "kubernetes.io/projected/1c53c174-8a70-4166-ab02-439d93723704-kube-api-access-dsctg") pod "network-check-target-bvktz" (UID: "1c53c174-8a70-4166-ab02-439d93723704") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:35:20.766640 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:20.766380 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:35:20.766860 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:20.766420 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:35:20.766860 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:20.766676 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bvktz" podUID="1c53c174-8a70-4166-ab02-439d93723704" Jul 10 20:35:20.766860 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:20.766441 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:35:20.766860 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:20.766742 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwvvg" podUID="93c55127-6ddc-4b15-844e-8b9585cc5d1e" Jul 10 20:35:20.767118 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:20.766881 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lmvbr" podUID="7da12ef6-d8c0-4c2c-823a-33c8d885e222" Jul 10 20:35:22.545851 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.545813 2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-64.ec2.internal" event="NodeReady" Jul 10 20:35:22.546259 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.545985 2530 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 20:35:22.589037 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.589004 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2fbhq"] Jul 10 20:35:22.591401 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.591369 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lz84m"] Jul 10 20:35:22.591401 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.591393 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2fbhq" Jul 10 20:35:22.593184 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.593161 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lz84m" Jul 10 20:35:22.594953 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.594934 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 20:35:22.595470 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.595444 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 20:35:22.595587 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.595519 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 20:35:22.596301 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.596281 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 20:35:22.596301 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.596294 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hp552\"" Jul 10 20:35:22.596428 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.596281 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 20:35:22.597734 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.597712 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j2m8z\"" Jul 10 20:35:22.603298 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.603275 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lz84m"] Jul 10 20:35:22.607254 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.607233 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2fbhq"] Jul 10 20:35:22.710227 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.710141 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2974e2ef-e853-4b53-8684-35405b1110fe-tmp-dir\") pod \"dns-default-2fbhq\" (UID: \"2974e2ef-e853-4b53-8684-35405b1110fe\") " pod="openshift-dns/dns-default-2fbhq" Jul 10 20:35:22.710227 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.710189 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxfgp\" (UniqueName: \"kubernetes.io/projected/2974e2ef-e853-4b53-8684-35405b1110fe-kube-api-access-bxfgp\") pod \"dns-default-2fbhq\" (UID: \"2974e2ef-e853-4b53-8684-35405b1110fe\") " pod="openshift-dns/dns-default-2fbhq" Jul 10 20:35:22.710433 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.710238 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf415b33-0aa4-49c9-a1bb-9f63108af200-cert\") pod \"ingress-canary-lz84m\" (UID: \"cf415b33-0aa4-49c9-a1bb-9f63108af200\") " pod="openshift-ingress-canary/ingress-canary-lz84m" Jul 10 20:35:22.710433 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.710261 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls\") pod \"dns-default-2fbhq\" (UID: \"2974e2ef-e853-4b53-8684-35405b1110fe\") " pod="openshift-dns/dns-default-2fbhq" Jul 10 20:35:22.710433 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.710278 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2974e2ef-e853-4b53-8684-35405b1110fe-config-volume\") pod \"dns-default-2fbhq\" (UID: \"2974e2ef-e853-4b53-8684-35405b1110fe\") " pod="openshift-dns/dns-default-2fbhq" Jul 10 20:35:22.710433 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.710319 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rwgs\" (UniqueName: \"kubernetes.io/projected/cf415b33-0aa4-49c9-a1bb-9f63108af200-kube-api-access-5rwgs\") pod \"ingress-canary-lz84m\" (UID: \"cf415b33-0aa4-49c9-a1bb-9f63108af200\") " pod="openshift-ingress-canary/ingress-canary-lz84m" Jul 10 20:35:22.767419 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.767386 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:35:22.767621 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.767386 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:35:22.767621 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.767386 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:35:22.769850 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.769810 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 20:35:22.769994 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.769915 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rvkjn\"" Jul 10 20:35:22.770202 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.770185 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rhdms\"" Jul 10 20:35:22.770278 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.770186 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 20:35:22.770363 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.770349 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 20:35:22.770521 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.770503 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 20:35:22.811159 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.811124 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf415b33-0aa4-49c9-a1bb-9f63108af200-cert\") pod \"ingress-canary-lz84m\" (UID: \"cf415b33-0aa4-49c9-a1bb-9f63108af200\") " pod="openshift-ingress-canary/ingress-canary-lz84m" Jul 10 20:35:22.811353 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.811172 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls\") pod \"dns-default-2fbhq\" (UID: \"2974e2ef-e853-4b53-8684-35405b1110fe\") " pod="openshift-dns/dns-default-2fbhq" Jul 10 20:35:22.811353 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.811206 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2974e2ef-e853-4b53-8684-35405b1110fe-config-volume\") pod \"dns-default-2fbhq\" (UID: \"2974e2ef-e853-4b53-8684-35405b1110fe\") " pod="openshift-dns/dns-default-2fbhq" Jul 10 20:35:22.811353 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.811267 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rwgs\" (UniqueName: \"kubernetes.io/projected/cf415b33-0aa4-49c9-a1bb-9f63108af200-kube-api-access-5rwgs\") pod \"ingress-canary-lz84m\" (UID: \"cf415b33-0aa4-49c9-a1bb-9f63108af200\") " pod="openshift-ingress-canary/ingress-canary-lz84m" Jul 10 20:35:22.811353 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.811305 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2974e2ef-e853-4b53-8684-35405b1110fe-tmp-dir\") pod \"dns-default-2fbhq\" (UID: \"2974e2ef-e853-4b53-8684-35405b1110fe\") " pod="openshift-dns/dns-default-2fbhq" Jul 10 20:35:22.811353 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.811337 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxfgp\" (UniqueName: \"kubernetes.io/projected/2974e2ef-e853-4b53-8684-35405b1110fe-kube-api-access-bxfgp\") pod \"dns-default-2fbhq\" (UID: \"2974e2ef-e853-4b53-8684-35405b1110fe\") " pod="openshift-dns/dns-default-2fbhq" Jul 10 20:35:22.811596 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:22.811358 2530 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 20:35:22.811596 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:22.811430 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls podName:2974e2ef-e853-4b53-8684-35405b1110fe nodeName:}" failed. No retries permitted until 2026-07-10 20:35:23.311413995 +0000 UTC m=+36.089995367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls") pod "dns-default-2fbhq" (UID: "2974e2ef-e853-4b53-8684-35405b1110fe") : secret "dns-default-metrics-tls" not found Jul 10 20:35:22.811911 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.811887 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2974e2ef-e853-4b53-8684-35405b1110fe-tmp-dir\") pod \"dns-default-2fbhq\" (UID: \"2974e2ef-e853-4b53-8684-35405b1110fe\") " pod="openshift-dns/dns-default-2fbhq" Jul 10 20:35:22.812161 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.812141 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2974e2ef-e853-4b53-8684-35405b1110fe-config-volume\") pod \"dns-default-2fbhq\" (UID: \"2974e2ef-e853-4b53-8684-35405b1110fe\") " pod="openshift-dns/dns-default-2fbhq" Jul 10 20:35:22.815790 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.815766 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf415b33-0aa4-49c9-a1bb-9f63108af200-cert\") pod \"ingress-canary-lz84m\" (UID: \"cf415b33-0aa4-49c9-a1bb-9f63108af200\") " pod="openshift-ingress-canary/ingress-canary-lz84m" Jul 10 20:35:22.822141 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.822114 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxfgp\" (UniqueName: \"kubernetes.io/projected/2974e2ef-e853-4b53-8684-35405b1110fe-kube-api-access-bxfgp\") pod \"dns-default-2fbhq\" (UID: \"2974e2ef-e853-4b53-8684-35405b1110fe\") " pod="openshift-dns/dns-default-2fbhq" Jul 10 20:35:22.822341 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.822320 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rwgs\" (UniqueName: \"kubernetes.io/projected/cf415b33-0aa4-49c9-a1bb-9f63108af200-kube-api-access-5rwgs\") pod \"ingress-canary-lz84m\" (UID: \"cf415b33-0aa4-49c9-a1bb-9f63108af200\") " pod="openshift-ingress-canary/ingress-canary-lz84m" Jul 10 20:35:22.911255 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:22.911222 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lz84m" Jul 10 20:35:23.316881 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:23.316826 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls\") pod \"dns-default-2fbhq\" (UID: \"2974e2ef-e853-4b53-8684-35405b1110fe\") " pod="openshift-dns/dns-default-2fbhq" Jul 10 20:35:23.317089 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:23.316969 2530 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 20:35:23.317089 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:23.317048 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls podName:2974e2ef-e853-4b53-8684-35405b1110fe nodeName:}" failed. No retries permitted until 2026-07-10 20:35:24.317029195 +0000 UTC m=+37.095610570 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls") pod "dns-default-2fbhq" (UID: "2974e2ef-e853-4b53-8684-35405b1110fe") : secret "dns-default-metrics-tls" not found Jul 10 20:35:24.330011 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:24.329300 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls\") pod \"dns-default-2fbhq\" (UID: \"2974e2ef-e853-4b53-8684-35405b1110fe\") " pod="openshift-dns/dns-default-2fbhq" Jul 10 20:35:24.330011 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:24.329517 2530 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 20:35:24.330011 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:24.329596 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls podName:2974e2ef-e853-4b53-8684-35405b1110fe nodeName:}" failed. No retries permitted until 2026-07-10 20:35:26.329575797 +0000 UTC m=+39.108157192 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls") pod "dns-default-2fbhq" (UID: "2974e2ef-e853-4b53-8684-35405b1110fe") : secret "dns-default-metrics-tls" not found Jul 10 20:35:24.392584 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:24.392346 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lz84m"] Jul 10 20:35:24.396637 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:35:24.396545 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf415b33_0aa4_49c9_a1bb_9f63108af200.slice/crio-c4f009cccbca5df08dcd5060cbb46f577dc8bad3f687f0726f84a52fc4657357 WatchSource:0}: Error finding container c4f009cccbca5df08dcd5060cbb46f577dc8bad3f687f0726f84a52fc4657357: Status 404 returned error can't find the container with id c4f009cccbca5df08dcd5060cbb46f577dc8bad3f687f0726f84a52fc4657357 Jul 10 20:35:25.054769 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:25.054734 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lz84m" event={"ID":"cf415b33-0aa4-49c9-a1bb-9f63108af200","Type":"ContainerStarted","Data":"c4f009cccbca5df08dcd5060cbb46f577dc8bad3f687f0726f84a52fc4657357"} Jul 10 20:35:25.057714 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:25.057682 2530 generic.go:356] "Generic (PLEG): container finished" podID="d6c0b1df-8049-472c-b375-1812f1ff620b" containerID="58ec5256a079fb879f2ae783ce1538c6ac5cfc0754d7bc1c14b1d1d1b568a01e" exitCode=0 Jul 10 20:35:25.057876 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:25.057720 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47wft" event={"ID":"d6c0b1df-8049-472c-b375-1812f1ff620b","Type":"ContainerDied","Data":"58ec5256a079fb879f2ae783ce1538c6ac5cfc0754d7bc1c14b1d1d1b568a01e"} Jul 10 20:35:26.063409 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:26.063372 2530 generic.go:356] "Generic (PLEG): container finished" podID="d6c0b1df-8049-472c-b375-1812f1ff620b" containerID="f01a90600f53a51278491fb31b8546eb92e7aa7bdf005af82c6ec24479850891" exitCode=0 Jul 10 20:35:26.064243 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:26.063446 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47wft" event={"ID":"d6c0b1df-8049-472c-b375-1812f1ff620b","Type":"ContainerDied","Data":"f01a90600f53a51278491fb31b8546eb92e7aa7bdf005af82c6ec24479850891"} Jul 10 20:35:26.344917 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:26.344814 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls\") pod \"dns-default-2fbhq\" (UID: \"2974e2ef-e853-4b53-8684-35405b1110fe\") " pod="openshift-dns/dns-default-2fbhq" Jul 10 20:35:26.345076 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:26.345026 2530 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 20:35:26.345129 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:26.345111 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls podName:2974e2ef-e853-4b53-8684-35405b1110fe nodeName:}" failed. No retries permitted until 2026-07-10 20:35:30.345090338 +0000 UTC m=+43.123671723 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls") pod "dns-default-2fbhq" (UID: "2974e2ef-e853-4b53-8684-35405b1110fe") : secret "dns-default-metrics-tls" not found Jul 10 20:35:27.069296 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:27.069264 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47wft" event={"ID":"d6c0b1df-8049-472c-b375-1812f1ff620b","Type":"ContainerStarted","Data":"ef95be7689a0be0cd3efb9b5cfcae08273c18f19963c0a9731adc214780fe8d5"} Jul 10 20:35:27.099293 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:27.099229 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-47wft" podStartSLOduration=7.737339267 podStartE2EDuration="39.099213485s" totalImagesPullingTime="31.361874218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:34:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:35:27.097761692 +0000 UTC m=+39.876343086" watchObservedRunningTime="2026-07-10 20:35:27.099213485 +0000 UTC m=+39.877794856" Jul 10 20:35:28.072822 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:28.072652 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lz84m" event={"ID":"cf415b33-0aa4-49c9-a1bb-9f63108af200","Type":"ContainerStarted","Data":"ef27fa6bf4b02557b128aac312f7e0457c12c62a106ff29e20d8860279a0e4cf"} Jul 10 20:35:28.087163 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:28.087113 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lz84m" podStartSLOduration=3.201590131 podStartE2EDuration="6.087098723s" totalImagesPullingTime="2.885508592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:35:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:35:28.08660462 +0000 UTC m=+40.865186014" watchObservedRunningTime="2026-07-10 20:35:28.087098723 +0000 UTC m=+40.865680118" Jul 10 20:35:30.376590 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:30.376552 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls\") pod \"dns-default-2fbhq\" (UID: \"2974e2ef-e853-4b53-8684-35405b1110fe\") " pod="openshift-dns/dns-default-2fbhq" Jul 10 20:35:30.377012 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:30.376666 2530 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 20:35:30.377012 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:30.376716 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls podName:2974e2ef-e853-4b53-8684-35405b1110fe nodeName:}" failed. No retries permitted until 2026-07-10 20:35:38.376703975 +0000 UTC m=+51.155285348 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls") pod "dns-default-2fbhq" (UID: "2974e2ef-e853-4b53-8684-35405b1110fe") : secret "dns-default-metrics-tls" not found Jul 10 20:35:38.429724 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:38.429682 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls\") pod \"dns-default-2fbhq\" (UID: \"2974e2ef-e853-4b53-8684-35405b1110fe\") " pod="openshift-dns/dns-default-2fbhq" Jul 10 20:35:38.430203 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:38.429823 2530 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 20:35:38.430203 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:38.429918 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls podName:2974e2ef-e853-4b53-8684-35405b1110fe nodeName:}" failed. No retries permitted until 2026-07-10 20:35:54.429899009 +0000 UTC m=+67.208480397 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls") pod "dns-default-2fbhq" (UID: "2974e2ef-e853-4b53-8684-35405b1110fe") : secret "dns-default-metrics-tls" not found Jul 10 20:35:49.054240 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:49.054202 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n7kg6" Jul 10 20:35:52.423584 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:52.423545 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7da12ef6-d8c0-4c2c-823a-33c8d885e222-original-pull-secret\") pod \"global-pull-secret-syncer-lmvbr\" (UID: \"7da12ef6-d8c0-4c2c-823a-33c8d885e222\") " pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:35:52.426475 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:52.426449 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 20:35:52.438188 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:52.438163 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7da12ef6-d8c0-4c2c-823a-33c8d885e222-original-pull-secret\") pod \"global-pull-secret-syncer-lmvbr\" (UID: \"7da12ef6-d8c0-4c2c-823a-33c8d885e222\") " pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:35:52.478329 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:52.478271 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lmvbr" Jul 10 20:35:52.524828 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:52.524786 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs\") pod \"network-metrics-daemon-dwvvg\" (UID: \"93c55127-6ddc-4b15-844e-8b9585cc5d1e\") " pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:35:52.527293 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:52.527271 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 20:35:52.535333 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:52.535301 2530 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 20:35:52.535497 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:52.535429 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs podName:93c55127-6ddc-4b15-844e-8b9585cc5d1e nodeName:}" failed. No retries permitted until 2026-07-10 20:36:56.535402129 +0000 UTC m=+129.313983517 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs") pod "network-metrics-daemon-dwvvg" (UID: "93c55127-6ddc-4b15-844e-8b9585cc5d1e") : secret "metrics-daemon-secret" not found Jul 10 20:35:52.602279 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:52.602245 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lmvbr"] Jul 10 20:35:52.605123 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:35:52.605091 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da12ef6_d8c0_4c2c_823a_33c8d885e222.slice/crio-12aa0fa777084c11fddc709836b9da43f0af37993a7924a7a7f948326471d591 WatchSource:0}: Error finding container 12aa0fa777084c11fddc709836b9da43f0af37993a7924a7a7f948326471d591: Status 404 returned error can't find the container with id 12aa0fa777084c11fddc709836b9da43f0af37993a7924a7a7f948326471d591 Jul 10 20:35:52.626352 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:52.626327 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsctg\" (UniqueName: \"kubernetes.io/projected/1c53c174-8a70-4166-ab02-439d93723704-kube-api-access-dsctg\") pod \"network-check-target-bvktz\" (UID: \"1c53c174-8a70-4166-ab02-439d93723704\") " pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:35:52.628684 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:52.628667 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 20:35:52.639642 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:52.639616 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 20:35:52.650100 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:52.650075 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsctg\" (UniqueName: \"kubernetes.io/projected/1c53c174-8a70-4166-ab02-439d93723704-kube-api-access-dsctg\") pod \"network-check-target-bvktz\" (UID: \"1c53c174-8a70-4166-ab02-439d93723704\") " pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:35:52.793314 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:52.793235 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rhdms\"" Jul 10 20:35:52.801490 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:52.801465 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:35:52.925655 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:52.925623 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bvktz"] Jul 10 20:35:52.930126 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:35:52.930097 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c53c174_8a70_4166_ab02_439d93723704.slice/crio-6524f0b5a5d2b4961bad769f73483f7216eb4cea10e4581ad68f82299749a4dd WatchSource:0}: Error finding container 6524f0b5a5d2b4961bad769f73483f7216eb4cea10e4581ad68f82299749a4dd: Status 404 returned error can't find the container with id 6524f0b5a5d2b4961bad769f73483f7216eb4cea10e4581ad68f82299749a4dd Jul 10 20:35:53.133263 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:53.133230 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bvktz" event={"ID":"1c53c174-8a70-4166-ab02-439d93723704","Type":"ContainerStarted","Data":"6524f0b5a5d2b4961bad769f73483f7216eb4cea10e4581ad68f82299749a4dd"} Jul 10 20:35:53.137210 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:53.134656 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lmvbr" event={"ID":"7da12ef6-d8c0-4c2c-823a-33c8d885e222","Type":"ContainerStarted","Data":"12aa0fa777084c11fddc709836b9da43f0af37993a7924a7a7f948326471d591"} Jul 10 20:35:54.443310 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:54.443265 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls\") pod \"dns-default-2fbhq\" (UID: \"2974e2ef-e853-4b53-8684-35405b1110fe\") " pod="openshift-dns/dns-default-2fbhq" Jul 10 20:35:54.443859 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:54.443455 2530 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 20:35:54.443859 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:35:54.443540 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls podName:2974e2ef-e853-4b53-8684-35405b1110fe nodeName:}" failed. No retries permitted until 2026-07-10 20:36:26.443521247 +0000 UTC m=+99.222102622 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls") pod "dns-default-2fbhq" (UID: "2974e2ef-e853-4b53-8684-35405b1110fe") : secret "dns-default-metrics-tls" not found Jul 10 20:35:58.149848 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:58.149794 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bvktz" event={"ID":"1c53c174-8a70-4166-ab02-439d93723704","Type":"ContainerStarted","Data":"d7d99fa57abf00c1934bfe097bc6ebd6179d0b42e32e4edd4aa4d9058ac948b9"} Jul 10 20:35:58.150273 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:58.149941 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:35:58.151297 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:58.151275 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lmvbr" event={"ID":"7da12ef6-d8c0-4c2c-823a-33c8d885e222","Type":"ContainerStarted","Data":"426deae99712a86986bc467160021c990e978c3a61901310ce72412db75bbae7"} Jul 10 20:35:58.168258 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:58.166256 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bvktz" podStartSLOduration=65.388967697 podStartE2EDuration="1m10.166235816s" totalImagesPullingTime="4.777268119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:35:58.164117094 +0000 UTC m=+70.942698488" watchObservedRunningTime="2026-07-10 20:35:58.166235816 +0000 UTC m=+70.944817211" Jul 10 20:35:58.178018 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:35:58.177966 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lmvbr" podStartSLOduration=65.062648366 podStartE2EDuration="1m10.177934722s" totalImagesPullingTime="5.115286356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:35:58.177622869 +0000 UTC m=+70.956204262" watchObservedRunningTime="2026-07-10 20:35:58.177934722 +0000 UTC m=+70.956516111" Jul 10 20:36:16.603924 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.603792 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt"] Jul 10 20:36:16.606362 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.606340 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f758d48-72rtz"] Jul 10 20:36:16.606496 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.606476 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" Jul 10 20:36:16.608479 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.608456 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f758d48-72rtz" Jul 10 20:36:16.608576 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.608545 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 10 20:36:16.608825 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.608804 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 20:36:16.609049 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.609033 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 10 20:36:16.609090 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.609060 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 10 20:36:16.609090 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.609063 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 20:36:16.609405 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.609366 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 20:36:16.609405 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.609397 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 10 20:36:16.610653 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.610633 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 10 20:36:16.618661 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.618635 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f758d48-72rtz"] Jul 10 20:36:16.620653 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.620629 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt"] Jul 10 20:36:16.694653 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.694608 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5293202-c7c9-47ba-9c7d-37bbad0668aa-tmp\") pod \"klusterlet-addon-workmgr-f758d48-72rtz\" (UID: \"c5293202-c7c9-47ba-9c7d-37bbad0668aa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f758d48-72rtz" Jul 10 20:36:16.694653 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.694647 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9a547e11-0f61-4327-90bd-bbe2db1ceb34-hub\") pod \"cluster-proxy-proxy-agent-846c4b5f54-vh4kt\" (UID: \"9a547e11-0f61-4327-90bd-bbe2db1ceb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" Jul 10 20:36:16.694884 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.694690 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9a547e11-0f61-4327-90bd-bbe2db1ceb34-ca\") pod \"cluster-proxy-proxy-agent-846c4b5f54-vh4kt\" (UID: \"9a547e11-0f61-4327-90bd-bbe2db1ceb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" Jul 10 20:36:16.694884 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.694740 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a547e11-0f61-4327-90bd-bbe2db1ceb34-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-846c4b5f54-vh4kt\" (UID: \"9a547e11-0f61-4327-90bd-bbe2db1ceb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" Jul 10 20:36:16.694884 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.694778 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a547e11-0f61-4327-90bd-bbe2db1ceb34-additional-service-ca\") pod \"cluster-proxy-proxy-agent-846c4b5f54-vh4kt\" (UID: \"9a547e11-0f61-4327-90bd-bbe2db1ceb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" Jul 10 20:36:16.694884 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.694807 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lj87\" (UniqueName: \"kubernetes.io/projected/c5293202-c7c9-47ba-9c7d-37bbad0668aa-kube-api-access-9lj87\") pod \"klusterlet-addon-workmgr-f758d48-72rtz\" (UID: \"c5293202-c7c9-47ba-9c7d-37bbad0668aa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f758d48-72rtz" Jul 10 20:36:16.694884 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.694826 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9a547e11-0f61-4327-90bd-bbe2db1ceb34-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-846c4b5f54-vh4kt\" (UID: \"9a547e11-0f61-4327-90bd-bbe2db1ceb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" Jul 10 20:36:16.695039 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.694890 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msdmq\" (UniqueName: \"kubernetes.io/projected/9a547e11-0f61-4327-90bd-bbe2db1ceb34-kube-api-access-msdmq\") pod \"cluster-proxy-proxy-agent-846c4b5f54-vh4kt\" (UID: \"9a547e11-0f61-4327-90bd-bbe2db1ceb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" Jul 10 20:36:16.695039 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.694927 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c5293202-c7c9-47ba-9c7d-37bbad0668aa-klusterlet-config\") pod \"klusterlet-addon-workmgr-f758d48-72rtz\" (UID: \"c5293202-c7c9-47ba-9c7d-37bbad0668aa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f758d48-72rtz" Jul 10 20:36:16.795578 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.795533 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5293202-c7c9-47ba-9c7d-37bbad0668aa-tmp\") pod \"klusterlet-addon-workmgr-f758d48-72rtz\" (UID: \"c5293202-c7c9-47ba-9c7d-37bbad0668aa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f758d48-72rtz" Jul 10 20:36:16.795578 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.795579 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9a547e11-0f61-4327-90bd-bbe2db1ceb34-hub\") pod \"cluster-proxy-proxy-agent-846c4b5f54-vh4kt\" (UID: \"9a547e11-0f61-4327-90bd-bbe2db1ceb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" Jul 10 20:36:16.795767 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.795609 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9a547e11-0f61-4327-90bd-bbe2db1ceb34-ca\") pod \"cluster-proxy-proxy-agent-846c4b5f54-vh4kt\" (UID: \"9a547e11-0f61-4327-90bd-bbe2db1ceb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" Jul 10 20:36:16.795767 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.795629 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a547e11-0f61-4327-90bd-bbe2db1ceb34-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-846c4b5f54-vh4kt\" (UID: \"9a547e11-0f61-4327-90bd-bbe2db1ceb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" Jul 10 20:36:16.795767 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.795646 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a547e11-0f61-4327-90bd-bbe2db1ceb34-additional-service-ca\") pod \"cluster-proxy-proxy-agent-846c4b5f54-vh4kt\" (UID: \"9a547e11-0f61-4327-90bd-bbe2db1ceb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" Jul 10 20:36:16.795767 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.795666 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lj87\" (UniqueName: \"kubernetes.io/projected/c5293202-c7c9-47ba-9c7d-37bbad0668aa-kube-api-access-9lj87\") pod \"klusterlet-addon-workmgr-f758d48-72rtz\" (UID: \"c5293202-c7c9-47ba-9c7d-37bbad0668aa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f758d48-72rtz" Jul 10 20:36:16.795767 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.795689 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9a547e11-0f61-4327-90bd-bbe2db1ceb34-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-846c4b5f54-vh4kt\" (UID: \"9a547e11-0f61-4327-90bd-bbe2db1ceb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" Jul 10 20:36:16.795767 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.795728 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msdmq\" (UniqueName: \"kubernetes.io/projected/9a547e11-0f61-4327-90bd-bbe2db1ceb34-kube-api-access-msdmq\") pod \"cluster-proxy-proxy-agent-846c4b5f54-vh4kt\" (UID: \"9a547e11-0f61-4327-90bd-bbe2db1ceb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" Jul 10 20:36:16.795767 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.795766 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c5293202-c7c9-47ba-9c7d-37bbad0668aa-klusterlet-config\") pod \"klusterlet-addon-workmgr-f758d48-72rtz\" (UID: \"c5293202-c7c9-47ba-9c7d-37bbad0668aa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f758d48-72rtz" Jul 10 20:36:16.796111 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.796027 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5293202-c7c9-47ba-9c7d-37bbad0668aa-tmp\") pod \"klusterlet-addon-workmgr-f758d48-72rtz\" (UID: \"c5293202-c7c9-47ba-9c7d-37bbad0668aa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f758d48-72rtz" Jul 10 20:36:16.796752 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.796723 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a547e11-0f61-4327-90bd-bbe2db1ceb34-additional-service-ca\") pod \"cluster-proxy-proxy-agent-846c4b5f54-vh4kt\" (UID: \"9a547e11-0f61-4327-90bd-bbe2db1ceb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" Jul 10 20:36:16.798357 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.798333 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9a547e11-0f61-4327-90bd-bbe2db1ceb34-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-846c4b5f54-vh4kt\" (UID: \"9a547e11-0f61-4327-90bd-bbe2db1ceb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" Jul 10 20:36:16.798451 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.798400 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9a547e11-0f61-4327-90bd-bbe2db1ceb34-hub\") pod \"cluster-proxy-proxy-agent-846c4b5f54-vh4kt\" (UID: \"9a547e11-0f61-4327-90bd-bbe2db1ceb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" Jul 10 20:36:16.798493 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.798465 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a547e11-0f61-4327-90bd-bbe2db1ceb34-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-846c4b5f54-vh4kt\" (UID: \"9a547e11-0f61-4327-90bd-bbe2db1ceb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" Jul 10 20:36:16.798559 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.798542 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9a547e11-0f61-4327-90bd-bbe2db1ceb34-ca\") pod \"cluster-proxy-proxy-agent-846c4b5f54-vh4kt\" (UID: \"9a547e11-0f61-4327-90bd-bbe2db1ceb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" Jul 10 20:36:16.798794 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.798774 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c5293202-c7c9-47ba-9c7d-37bbad0668aa-klusterlet-config\") pod \"klusterlet-addon-workmgr-f758d48-72rtz\" (UID: \"c5293202-c7c9-47ba-9c7d-37bbad0668aa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f758d48-72rtz" Jul 10 20:36:16.803255 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.803235 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msdmq\" (UniqueName: \"kubernetes.io/projected/9a547e11-0f61-4327-90bd-bbe2db1ceb34-kube-api-access-msdmq\") pod \"cluster-proxy-proxy-agent-846c4b5f54-vh4kt\" (UID: \"9a547e11-0f61-4327-90bd-bbe2db1ceb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" Jul 10 20:36:16.803513 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.803493 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lj87\" (UniqueName: \"kubernetes.io/projected/c5293202-c7c9-47ba-9c7d-37bbad0668aa-kube-api-access-9lj87\") pod \"klusterlet-addon-workmgr-f758d48-72rtz\" (UID: \"c5293202-c7c9-47ba-9c7d-37bbad0668aa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f758d48-72rtz" Jul 10 20:36:16.923458 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.923420 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" Jul 10 20:36:16.928300 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:16.928280 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f758d48-72rtz" Jul 10 20:36:17.063432 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:17.063395 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt"] Jul 10 20:36:17.067065 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:36:17.067040 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a547e11_0f61_4327_90bd_bbe2db1ceb34.slice/crio-2820427b0eaef72163e2ff1b0f3e85788ae58595f2bff1ff893388b2f39b54df WatchSource:0}: Error finding container 2820427b0eaef72163e2ff1b0f3e85788ae58595f2bff1ff893388b2f39b54df: Status 404 returned error can't find the container with id 2820427b0eaef72163e2ff1b0f3e85788ae58595f2bff1ff893388b2f39b54df Jul 10 20:36:17.071503 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:17.071480 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f758d48-72rtz"] Jul 10 20:36:17.074448 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:36:17.074422 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5293202_c7c9_47ba_9c7d_37bbad0668aa.slice/crio-2b10277a31dcc9e58bca04776c28f43fca16c2ad1f46f632f248aacb7d1ed073 WatchSource:0}: Error finding container 2b10277a31dcc9e58bca04776c28f43fca16c2ad1f46f632f248aacb7d1ed073: Status 404 returned error can't find the container with id 2b10277a31dcc9e58bca04776c28f43fca16c2ad1f46f632f248aacb7d1ed073 Jul 10 20:36:17.198598 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:17.198510 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" event={"ID":"9a547e11-0f61-4327-90bd-bbe2db1ceb34","Type":"ContainerStarted","Data":"2820427b0eaef72163e2ff1b0f3e85788ae58595f2bff1ff893388b2f39b54df"} Jul 10 20:36:17.199438 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:17.199416 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f758d48-72rtz" event={"ID":"c5293202-c7c9-47ba-9c7d-37bbad0668aa","Type":"ContainerStarted","Data":"2b10277a31dcc9e58bca04776c28f43fca16c2ad1f46f632f248aacb7d1ed073"} Jul 10 20:36:22.220775 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:22.220709 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f758d48-72rtz" event={"ID":"c5293202-c7c9-47ba-9c7d-37bbad0668aa","Type":"ContainerStarted","Data":"3a6eb6593f3b255f4b6c67c179007165e313a3fca11f3a1dd2f8c1849d054cc5"} Jul 10 20:36:22.221240 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:22.220865 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f758d48-72rtz" Jul 10 20:36:22.222317 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:22.222299 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f758d48-72rtz" Jul 10 20:36:22.222710 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:22.222684 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" event={"ID":"9a547e11-0f61-4327-90bd-bbe2db1ceb34","Type":"ContainerStarted","Data":"beb43bc03c8a83d1223c6553778b35866797f8817717095055862b202ff030c0"} Jul 10 20:36:22.222785 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:22.222727 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" event={"ID":"9a547e11-0f61-4327-90bd-bbe2db1ceb34","Type":"ContainerStarted","Data":"35b446bd02071d7b7385637bf24823c8811e637606fce3430d0965b0709367bf"} Jul 10 20:36:22.235471 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:22.235419 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f758d48-72rtz" podStartSLOduration=1.264070136 podStartE2EDuration="6.235398824s" totalImagesPullingTime="4.971328688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:36:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:36:22.234438194 +0000 UTC m=+95.013019587" watchObservedRunningTime="2026-07-10 20:36:22.235398824 +0000 UTC m=+95.013980218" Jul 10 20:36:23.228892 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:23.228852 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" event={"ID":"9a547e11-0f61-4327-90bd-bbe2db1ceb34","Type":"ContainerStarted","Data":"82f60a2712c052a612d96eb79e410549ba5e7034a9c4db13d5808d90d3be0876"} Jul 10 20:36:23.247787 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:23.247716 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" podStartSLOduration=2.271333931 podStartE2EDuration="7.247697562s" totalImagesPullingTime="4.976363631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:36:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:36:23.246782502 +0000 UTC m=+96.025363897" watchObservedRunningTime="2026-07-10 20:36:23.247697562 +0000 UTC m=+96.026278957" Jul 10 20:36:26.471800 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:26.471756 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls\") pod \"dns-default-2fbhq\" (UID: \"2974e2ef-e853-4b53-8684-35405b1110fe\") " pod="openshift-dns/dns-default-2fbhq" Jul 10 20:36:26.472317 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:36:26.471932 2530 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 20:36:26.472317 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:36:26.472022 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls podName:2974e2ef-e853-4b53-8684-35405b1110fe nodeName:}" failed. No retries permitted until 2026-07-10 20:37:30.472000871 +0000 UTC m=+163.250582262 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls") pod "dns-default-2fbhq" (UID: "2974e2ef-e853-4b53-8684-35405b1110fe") : secret "dns-default-metrics-tls" not found Jul 10 20:36:29.155709 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:29.155668 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bvktz" Jul 10 20:36:56.583030 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:36:56.582993 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs\") pod \"network-metrics-daemon-dwvvg\" (UID: \"93c55127-6ddc-4b15-844e-8b9585cc5d1e\") " pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:36:56.583522 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:36:56.583140 2530 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 20:36:56.583522 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:36:56.583206 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs podName:93c55127-6ddc-4b15-844e-8b9585cc5d1e nodeName:}" failed. No retries permitted until 2026-07-10 20:38:58.583190326 +0000 UTC m=+251.361771698 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs") pod "network-metrics-daemon-dwvvg" (UID: "93c55127-6ddc-4b15-844e-8b9585cc5d1e") : secret "metrics-daemon-secret" not found Jul 10 20:37:25.605125 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:37:25.605078 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-2fbhq" podUID="2974e2ef-e853-4b53-8684-35405b1110fe" Jul 10 20:37:25.786129 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:37:25.786090 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-dwvvg" podUID="93c55127-6ddc-4b15-844e-8b9585cc5d1e" Jul 10 20:37:26.407314 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:37:26.407286 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2fbhq" Jul 10 20:37:30.521546 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:37:30.521503 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls\") pod \"dns-default-2fbhq\" (UID: \"2974e2ef-e853-4b53-8684-35405b1110fe\") " pod="openshift-dns/dns-default-2fbhq" Jul 10 20:37:30.521982 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:37:30.521628 2530 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 20:37:30.521982 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:37:30.521684 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls podName:2974e2ef-e853-4b53-8684-35405b1110fe nodeName:}" failed. No retries permitted until 2026-07-10 20:39:32.521666719 +0000 UTC m=+285.300248091 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls") pod "dns-default-2fbhq" (UID: "2974e2ef-e853-4b53-8684-35405b1110fe") : secret "dns-default-metrics-tls" not found Jul 10 20:37:40.766659 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:37:40.766560 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:38:22.221219 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:22.221160 2530 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f758d48-72rtz" podUID="c5293202-c7c9-47ba-9c7d-37bbad0668aa" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.9:8000/readyz\": dial tcp 10.133.0.9:8000: connect: connection refused" Jul 10 20:38:22.576988 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:22.576902 2530 generic.go:356] "Generic (PLEG): container finished" podID="c5293202-c7c9-47ba-9c7d-37bbad0668aa" containerID="3a6eb6593f3b255f4b6c67c179007165e313a3fca11f3a1dd2f8c1849d054cc5" exitCode=1 Jul 10 20:38:22.576988 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:22.576958 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f758d48-72rtz" event={"ID":"c5293202-c7c9-47ba-9c7d-37bbad0668aa","Type":"ContainerDied","Data":"3a6eb6593f3b255f4b6c67c179007165e313a3fca11f3a1dd2f8c1849d054cc5"} Jul 10 20:38:22.577292 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:22.577278 2530 scope.go:117] "RemoveContainer" containerID="3a6eb6593f3b255f4b6c67c179007165e313a3fca11f3a1dd2f8c1849d054cc5" Jul 10 20:38:23.581151 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:23.581115 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f758d48-72rtz" event={"ID":"c5293202-c7c9-47ba-9c7d-37bbad0668aa","Type":"ContainerStarted","Data":"50be59a1e56db8145be15afbbdf6f9fb41bb0fe1861d02f110fe52772955b03d"} Jul 10 20:38:23.581546 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:23.581406 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f758d48-72rtz" Jul 10 20:38:23.582016 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:23.581993 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f758d48-72rtz" Jul 10 20:38:49.348046 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:49.348008 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jspvl"] Jul 10 20:38:49.350030 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:49.350009 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:38:49.352329 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:49.352307 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 20:38:49.352447 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:49.352342 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n7x97\"" Jul 10 20:38:49.353080 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:49.353060 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 20:38:49.353168 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:49.353077 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 20:38:49.353168 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:49.353060 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 20:38:49.362171 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:49.362146 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jspvl"] Jul 10 20:38:49.467554 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:49.467515 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jspvl\" (UID: \"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5\") " pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:38:49.467554 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:49.467551 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsx29\" (UniqueName: \"kubernetes.io/projected/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-kube-api-access-hsx29\") pod \"insights-runtime-extractor-jspvl\" (UID: \"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5\") " pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:38:49.467770 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:49.467603 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jspvl\" (UID: \"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5\") " pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:38:49.467770 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:49.467636 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-crio-socket\") pod \"insights-runtime-extractor-jspvl\" (UID: \"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5\") " pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:38:49.467770 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:49.467693 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-data-volume\") pod \"insights-runtime-extractor-jspvl\" (UID: \"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5\") " pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:38:49.568100 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:49.568051 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-data-volume\") pod \"insights-runtime-extractor-jspvl\" (UID: \"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5\") " pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:38:49.568277 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:49.568114 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jspvl\" (UID: \"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5\") " pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:38:49.568277 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:49.568141 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsx29\" (UniqueName: \"kubernetes.io/projected/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-kube-api-access-hsx29\") pod \"insights-runtime-extractor-jspvl\" (UID: \"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5\") " pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:38:49.568277 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:49.568178 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jspvl\" (UID: \"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5\") " pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:38:49.568277 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:49.568202 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-crio-socket\") pod \"insights-runtime-extractor-jspvl\" (UID: \"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5\") " pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:38:49.568277 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:38:49.568253 2530 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 20:38:49.568451 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:38:49.568296 2530 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 20:38:49.568451 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:49.568296 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-crio-socket\") pod \"insights-runtime-extractor-jspvl\" (UID: \"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5\") " pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:38:49.568451 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:38:49.568337 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-kube-rbac-proxy-cm podName:eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5 nodeName:}" failed. No retries permitted until 2026-07-10 20:38:50.068316672 +0000 UTC m=+242.846898067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jspvl" (UID: "eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5") : configmap "kube-rbac-proxy" not found Jul 10 20:38:49.568451 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:38:49.568361 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-insights-runtime-extractor-tls podName:eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5 nodeName:}" failed. No retries permitted until 2026-07-10 20:38:50.068347695 +0000 UTC m=+242.846929067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jspvl" (UID: "eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5") : secret "insights-runtime-extractor-tls" not found Jul 10 20:38:49.568591 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:49.568446 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-data-volume\") pod \"insights-runtime-extractor-jspvl\" (UID: \"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5\") " pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:38:49.576356 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:49.576323 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsx29\" (UniqueName: \"kubernetes.io/projected/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-kube-api-access-hsx29\") pod \"insights-runtime-extractor-jspvl\" (UID: \"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5\") " pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:38:50.072685 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:50.072645 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jspvl\" (UID: \"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5\") " pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:38:50.072685 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:38:50.072687 2530 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 20:38:50.072943 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:50.072717 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jspvl\" (UID: \"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5\") " pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:38:50.072943 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:38:50.072817 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-kube-rbac-proxy-cm podName:eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5 nodeName:}" failed. No retries permitted until 2026-07-10 20:38:51.072801886 +0000 UTC m=+243.851383259 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jspvl" (UID: "eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5") : configmap "kube-rbac-proxy" not found Jul 10 20:38:50.072943 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:38:50.072885 2530 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 20:38:50.073048 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:38:50.072966 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-insights-runtime-extractor-tls podName:eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5 nodeName:}" failed. No retries permitted until 2026-07-10 20:38:51.072949628 +0000 UTC m=+243.851531007 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jspvl" (UID: "eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5") : secret "insights-runtime-extractor-tls" not found Jul 10 20:38:51.078476 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:51.078430 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jspvl\" (UID: \"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5\") " pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:38:51.078894 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:51.078544 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jspvl\" (UID: \"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5\") " pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:38:51.078894 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:38:51.078594 2530 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 20:38:51.078894 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:38:51.078609 2530 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 20:38:51.078894 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:38:51.078663 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-kube-rbac-proxy-cm podName:eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5 nodeName:}" failed. No retries permitted until 2026-07-10 20:38:53.07864615 +0000 UTC m=+245.857227521 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jspvl" (UID: "eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5") : configmap "kube-rbac-proxy" not found Jul 10 20:38:51.078894 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:38:51.078679 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-insights-runtime-extractor-tls podName:eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5 nodeName:}" failed. No retries permitted until 2026-07-10 20:38:53.078671087 +0000 UTC m=+245.857252459 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jspvl" (UID: "eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5") : secret "insights-runtime-extractor-tls" not found Jul 10 20:38:51.306722 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:51.306696 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kpwbz_12401ef9-0e51-4dc0-8e83-ccb7fe160a6e/dns-node-resolver/0.log" Jul 10 20:38:52.307273 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:52.307249 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w9q99_7b2743f9-3a33-4dfa-a239-dbd0c983bd20/node-ca/0.log" Jul 10 20:38:53.094342 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:53.094302 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jspvl\" (UID: \"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5\") " pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:38:53.094342 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:53.094351 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jspvl\" (UID: \"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5\") " pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:38:53.094585 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:38:53.094429 2530 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 20:38:53.094585 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:38:53.094467 2530 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 20:38:53.094585 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:38:53.094497 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-kube-rbac-proxy-cm podName:eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5 nodeName:}" failed. No retries permitted until 2026-07-10 20:38:57.094482792 +0000 UTC m=+249.873064164 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jspvl" (UID: "eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5") : configmap "kube-rbac-proxy" not found Jul 10 20:38:53.094585 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:38:53.094511 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-insights-runtime-extractor-tls podName:eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5 nodeName:}" failed. No retries permitted until 2026-07-10 20:38:57.094505501 +0000 UTC m=+249.873086873 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jspvl" (UID: "eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5") : secret "insights-runtime-extractor-tls" not found Jul 10 20:38:53.307947 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:53.307921 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lz84m_cf415b33-0aa4-49c9-a1bb-9f63108af200/serve-healthcheck-canary/0.log" Jul 10 20:38:57.126961 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:57.126920 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jspvl\" (UID: \"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5\") " pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:38:57.127353 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:57.126973 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jspvl\" (UID: \"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5\") " pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:38:57.127353 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:38:57.127038 2530 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 20:38:57.127353 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:38:57.127094 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-kube-rbac-proxy-cm podName:eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5 nodeName:}" failed. No retries permitted until 2026-07-10 20:39:05.127080077 +0000 UTC m=+257.905661448 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jspvl" (UID: "eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5") : configmap "kube-rbac-proxy" not found Jul 10 20:38:57.127353 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:38:57.127097 2530 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 20:38:57.127353 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:38:57.127136 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-insights-runtime-extractor-tls podName:eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5 nodeName:}" failed. No retries permitted until 2026-07-10 20:39:05.127125487 +0000 UTC m=+257.905706858 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jspvl" (UID: "eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5") : secret "insights-runtime-extractor-tls" not found Jul 10 20:38:58.639766 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:38:58.639714 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs\") pod \"network-metrics-daemon-dwvvg\" (UID: \"93c55127-6ddc-4b15-844e-8b9585cc5d1e\") " pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:38:58.640182 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:38:58.639885 2530 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 20:38:58.640182 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:38:58.639946 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs podName:93c55127-6ddc-4b15-844e-8b9585cc5d1e nodeName:}" failed. No retries permitted until 2026-07-10 20:41:00.639930859 +0000 UTC m=+373.418512230 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs") pod "network-metrics-daemon-dwvvg" (UID: "93c55127-6ddc-4b15-844e-8b9585cc5d1e") : secret "metrics-daemon-secret" not found Jul 10 20:39:05.192803 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:05.192746 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jspvl\" (UID: \"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5\") " pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:39:05.192803 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:05.192816 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jspvl\" (UID: \"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5\") " pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:39:05.193343 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:05.193309 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jspvl\" (UID: \"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5\") " pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:39:05.195122 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:05.195099 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jspvl\" (UID: \"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5\") " pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:39:05.259163 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:05.259123 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jspvl" Jul 10 20:39:05.388358 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:05.388325 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jspvl"] Jul 10 20:39:05.392104 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:39:05.392072 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3d7e6c_15fa_4c26_b7c2_7282aa5e71e5.slice/crio-e01007173ea3e02b17f135bf1490ae6166de1038711474fe570a1c18ca467f48 WatchSource:0}: Error finding container e01007173ea3e02b17f135bf1490ae6166de1038711474fe570a1c18ca467f48: Status 404 returned error can't find the container with id e01007173ea3e02b17f135bf1490ae6166de1038711474fe570a1c18ca467f48 Jul 10 20:39:05.703438 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:05.703406 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jspvl" event={"ID":"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5","Type":"ContainerStarted","Data":"e01007173ea3e02b17f135bf1490ae6166de1038711474fe570a1c18ca467f48"} Jul 10 20:39:09.717968 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:09.717926 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jspvl" event={"ID":"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5","Type":"ContainerStarted","Data":"387cdbf74ed4ea1e75c2125607ac71778da600ad650720cd39edc1fabc3e86ce"} Jul 10 20:39:10.723559 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:10.723526 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jspvl" event={"ID":"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5","Type":"ContainerStarted","Data":"70b6863974dc233d2b784807e8efaf5d6c895268a74f15ef34fbdf55623fbf3e"} Jul 10 20:39:12.731434 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:12.731396 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jspvl" event={"ID":"eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5","Type":"ContainerStarted","Data":"6c4cd535d13212889770bb13c79d30eedaf2d2a756c50ee1d77908ebf08f65ca"} Jul 10 20:39:12.748888 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:12.748816 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jspvl" podStartSLOduration=16.88741694 podStartE2EDuration="23.74880021s" totalImagesPullingTime="6.86138327s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:38:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:39:12.748673566 +0000 UTC m=+265.527254960" watchObservedRunningTime="2026-07-10 20:39:12.74880021 +0000 UTC m=+265.527381604" Jul 10 20:39:17.943645 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:17.943614 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fbdd9bbdd-mxcqx"] Jul 10 20:39:17.946477 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:17.946460 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbdd9bbdd-mxcqx" Jul 10 20:39:17.948860 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:17.948822 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 10 20:39:17.949561 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:17.949541 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 10 20:39:17.949694 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:17.949641 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hnc4g\"" Jul 10 20:39:17.949694 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:17.949679 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 10 20:39:17.949824 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:17.949741 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 20:39:17.949824 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:17.949769 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 10 20:39:17.950016 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:17.949980 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 20:39:17.950098 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:17.950053 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 10 20:39:17.957055 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:17.957031 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fbdd9bbdd-mxcqx"] Jul 10 20:39:17.992334 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:17.992300 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73da068e-4563-4406-a360-44b01cb07911-oauth-serving-cert\") pod \"console-7fbdd9bbdd-mxcqx\" (UID: \"73da068e-4563-4406-a360-44b01cb07911\") " pod="openshift-console/console-7fbdd9bbdd-mxcqx" Jul 10 20:39:17.992334 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:17.992337 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fm88\" (UniqueName: \"kubernetes.io/projected/73da068e-4563-4406-a360-44b01cb07911-kube-api-access-6fm88\") pod \"console-7fbdd9bbdd-mxcqx\" (UID: \"73da068e-4563-4406-a360-44b01cb07911\") " pod="openshift-console/console-7fbdd9bbdd-mxcqx" Jul 10 20:39:17.992572 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:17.992362 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73da068e-4563-4406-a360-44b01cb07911-console-oauth-config\") pod \"console-7fbdd9bbdd-mxcqx\" (UID: \"73da068e-4563-4406-a360-44b01cb07911\") " pod="openshift-console/console-7fbdd9bbdd-mxcqx" Jul 10 20:39:17.992572 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:17.992382 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73da068e-4563-4406-a360-44b01cb07911-console-serving-cert\") pod \"console-7fbdd9bbdd-mxcqx\" (UID: \"73da068e-4563-4406-a360-44b01cb07911\") " pod="openshift-console/console-7fbdd9bbdd-mxcqx" Jul 10 20:39:17.992572 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:17.992396 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73da068e-4563-4406-a360-44b01cb07911-service-ca\") pod \"console-7fbdd9bbdd-mxcqx\" (UID: \"73da068e-4563-4406-a360-44b01cb07911\") " pod="openshift-console/console-7fbdd9bbdd-mxcqx" Jul 10 20:39:17.992572 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:17.992450 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73da068e-4563-4406-a360-44b01cb07911-console-config\") pod \"console-7fbdd9bbdd-mxcqx\" (UID: \"73da068e-4563-4406-a360-44b01cb07911\") " pod="openshift-console/console-7fbdd9bbdd-mxcqx" Jul 10 20:39:18.092987 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:18.092932 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73da068e-4563-4406-a360-44b01cb07911-oauth-serving-cert\") pod \"console-7fbdd9bbdd-mxcqx\" (UID: \"73da068e-4563-4406-a360-44b01cb07911\") " pod="openshift-console/console-7fbdd9bbdd-mxcqx" Jul 10 20:39:18.092987 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:18.092984 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fm88\" (UniqueName: \"kubernetes.io/projected/73da068e-4563-4406-a360-44b01cb07911-kube-api-access-6fm88\") pod \"console-7fbdd9bbdd-mxcqx\" (UID: \"73da068e-4563-4406-a360-44b01cb07911\") " pod="openshift-console/console-7fbdd9bbdd-mxcqx" Jul 10 20:39:18.092987 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:18.093003 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73da068e-4563-4406-a360-44b01cb07911-console-oauth-config\") pod \"console-7fbdd9bbdd-mxcqx\" (UID: \"73da068e-4563-4406-a360-44b01cb07911\") " pod="openshift-console/console-7fbdd9bbdd-mxcqx" Jul 10 20:39:18.093289 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:18.093019 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73da068e-4563-4406-a360-44b01cb07911-console-serving-cert\") pod \"console-7fbdd9bbdd-mxcqx\" (UID: \"73da068e-4563-4406-a360-44b01cb07911\") " pod="openshift-console/console-7fbdd9bbdd-mxcqx" Jul 10 20:39:18.093289 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:18.093035 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73da068e-4563-4406-a360-44b01cb07911-service-ca\") pod \"console-7fbdd9bbdd-mxcqx\" (UID: \"73da068e-4563-4406-a360-44b01cb07911\") " pod="openshift-console/console-7fbdd9bbdd-mxcqx" Jul 10 20:39:18.093289 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:18.093051 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73da068e-4563-4406-a360-44b01cb07911-console-config\") pod \"console-7fbdd9bbdd-mxcqx\" (UID: \"73da068e-4563-4406-a360-44b01cb07911\") " pod="openshift-console/console-7fbdd9bbdd-mxcqx" Jul 10 20:39:18.093705 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:18.093675 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73da068e-4563-4406-a360-44b01cb07911-console-config\") pod \"console-7fbdd9bbdd-mxcqx\" (UID: \"73da068e-4563-4406-a360-44b01cb07911\") " pod="openshift-console/console-7fbdd9bbdd-mxcqx" Jul 10 20:39:18.093821 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:18.093747 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73da068e-4563-4406-a360-44b01cb07911-oauth-serving-cert\") pod \"console-7fbdd9bbdd-mxcqx\" (UID: \"73da068e-4563-4406-a360-44b01cb07911\") " pod="openshift-console/console-7fbdd9bbdd-mxcqx" Jul 10 20:39:18.093821 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:18.093760 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73da068e-4563-4406-a360-44b01cb07911-service-ca\") pod \"console-7fbdd9bbdd-mxcqx\" (UID: \"73da068e-4563-4406-a360-44b01cb07911\") " pod="openshift-console/console-7fbdd9bbdd-mxcqx" Jul 10 20:39:18.095432 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:18.095412 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73da068e-4563-4406-a360-44b01cb07911-console-serving-cert\") pod \"console-7fbdd9bbdd-mxcqx\" (UID: \"73da068e-4563-4406-a360-44b01cb07911\") " pod="openshift-console/console-7fbdd9bbdd-mxcqx" Jul 10 20:39:18.095512 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:18.095492 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73da068e-4563-4406-a360-44b01cb07911-console-oauth-config\") pod \"console-7fbdd9bbdd-mxcqx\" (UID: \"73da068e-4563-4406-a360-44b01cb07911\") " pod="openshift-console/console-7fbdd9bbdd-mxcqx" Jul 10 20:39:18.101164 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:18.101138 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fm88\" (UniqueName: \"kubernetes.io/projected/73da068e-4563-4406-a360-44b01cb07911-kube-api-access-6fm88\") pod \"console-7fbdd9bbdd-mxcqx\" (UID: \"73da068e-4563-4406-a360-44b01cb07911\") " pod="openshift-console/console-7fbdd9bbdd-mxcqx" Jul 10 20:39:18.254860 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:18.254758 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbdd9bbdd-mxcqx" Jul 10 20:39:18.382355 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:18.382322 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fbdd9bbdd-mxcqx"] Jul 10 20:39:18.386530 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:39:18.386486 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73da068e_4563_4406_a360_44b01cb07911.slice/crio-2d7589ea85aaafe0c159043e150614dab1c7b90a5dc15ebcb841507ccf0b4124 WatchSource:0}: Error finding container 2d7589ea85aaafe0c159043e150614dab1c7b90a5dc15ebcb841507ccf0b4124: Status 404 returned error can't find the container with id 2d7589ea85aaafe0c159043e150614dab1c7b90a5dc15ebcb841507ccf0b4124 Jul 10 20:39:18.749519 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:18.749485 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbdd9bbdd-mxcqx" event={"ID":"73da068e-4563-4406-a360-44b01cb07911","Type":"ContainerStarted","Data":"2d7589ea85aaafe0c159043e150614dab1c7b90a5dc15ebcb841507ccf0b4124"} Jul 10 20:39:19.676250 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:19.676216 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zzmxl"] Jul 10 20:39:19.679017 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:19.679000 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zzmxl" Jul 10 20:39:19.684169 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:19.684147 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 10 20:39:19.684340 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:19.684324 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-l8ccj\"" Jul 10 20:39:19.688885 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:19.688860 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zzmxl"] Jul 10 20:39:19.706489 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:19.706452 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f279cd39-20ee-46df-8f56-13e2b6fe82e8-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-zzmxl\" (UID: \"f279cd39-20ee-46df-8f56-13e2b6fe82e8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zzmxl" Jul 10 20:39:19.807384 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:19.807349 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f279cd39-20ee-46df-8f56-13e2b6fe82e8-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-zzmxl\" (UID: \"f279cd39-20ee-46df-8f56-13e2b6fe82e8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zzmxl" Jul 10 20:39:19.807573 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:39:19.807505 2530 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 10 20:39:19.807618 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:39:19.807598 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f279cd39-20ee-46df-8f56-13e2b6fe82e8-tls-certificates podName:f279cd39-20ee-46df-8f56-13e2b6fe82e8 nodeName:}" failed. No retries permitted until 2026-07-10 20:39:20.307576345 +0000 UTC m=+273.086157720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/f279cd39-20ee-46df-8f56-13e2b6fe82e8-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-zzmxl" (UID: "f279cd39-20ee-46df-8f56-13e2b6fe82e8") : secret "prometheus-operator-admission-webhook-tls" not found Jul 10 20:39:20.311316 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:20.311265 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f279cd39-20ee-46df-8f56-13e2b6fe82e8-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-zzmxl\" (UID: \"f279cd39-20ee-46df-8f56-13e2b6fe82e8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zzmxl" Jul 10 20:39:20.314146 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:20.314121 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f279cd39-20ee-46df-8f56-13e2b6fe82e8-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-zzmxl\" (UID: \"f279cd39-20ee-46df-8f56-13e2b6fe82e8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zzmxl" Jul 10 20:39:20.588384 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:20.588298 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zzmxl" Jul 10 20:39:21.579310 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:21.579285 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zzmxl"] Jul 10 20:39:21.582886 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:39:21.582824 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf279cd39_20ee_46df_8f56_13e2b6fe82e8.slice/crio-2e7ff9250df3c82ee30172f79d59b499ee74c6a55bcfe3de3f38cc1f540c2bbb WatchSource:0}: Error finding container 2e7ff9250df3c82ee30172f79d59b499ee74c6a55bcfe3de3f38cc1f540c2bbb: Status 404 returned error can't find the container with id 2e7ff9250df3c82ee30172f79d59b499ee74c6a55bcfe3de3f38cc1f540c2bbb Jul 10 20:39:21.761414 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:21.761318 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbdd9bbdd-mxcqx" event={"ID":"73da068e-4563-4406-a360-44b01cb07911","Type":"ContainerStarted","Data":"010ab227998c8f4549f8b4bf2fef2985e94918b22e8d71a100bc63139daa5d70"} Jul 10 20:39:21.762499 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:21.762474 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zzmxl" event={"ID":"f279cd39-20ee-46df-8f56-13e2b6fe82e8","Type":"ContainerStarted","Data":"2e7ff9250df3c82ee30172f79d59b499ee74c6a55bcfe3de3f38cc1f540c2bbb"} Jul 10 20:39:21.781250 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:21.781090 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7fbdd9bbdd-mxcqx" podStartSLOduration=1.658441925 podStartE2EDuration="4.781070512s" totalImagesPullingTime="3.122628587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:39:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:39:21.780308145 +0000 UTC m=+274.558889533" watchObservedRunningTime="2026-07-10 20:39:21.781070512 +0000 UTC m=+274.559651907" Jul 10 20:39:22.766934 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:22.766887 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zzmxl" event={"ID":"f279cd39-20ee-46df-8f56-13e2b6fe82e8","Type":"ContainerStarted","Data":"a21eb09a8e55a91fe18656d8843a9a34c3391b7267f4d188b5426cc8f6b9f0c6"} Jul 10 20:39:22.784665 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:22.784614 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zzmxl" podStartSLOduration=2.750584514 podStartE2EDuration="3.784598814s" totalImagesPullingTime="1.0340143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:39:22.783759125 +0000 UTC m=+275.562340522" watchObservedRunningTime="2026-07-10 20:39:22.784598814 +0000 UTC m=+275.563180187" Jul 10 20:39:23.770309 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:23.770271 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zzmxl" Jul 10 20:39:23.774743 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:23.774721 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zzmxl" Jul 10 20:39:28.255779 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:28.255729 2530 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fbdd9bbdd-mxcqx" Jul 10 20:39:28.256212 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:28.255856 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fbdd9bbdd-mxcqx" Jul 10 20:39:28.257086 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:28.257062 2530 patch_prober.go:28] interesting pod/console-7fbdd9bbdd-mxcqx container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.11:8443/health\": dial tcp 10.133.0.11:8443: connect: connection refused" start-of-body= Jul 10 20:39:28.257208 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:28.257106 2530 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7fbdd9bbdd-mxcqx" podUID="73da068e-4563-4406-a360-44b01cb07911" containerName="console" probeResult="failure" output="Get \"https://10.133.0.11:8443/health\": dial tcp 10.133.0.11:8443: connect: connection refused" Jul 10 20:39:29.408281 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:39:29.408235 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-2fbhq" podUID="2974e2ef-e853-4b53-8684-35405b1110fe" Jul 10 20:39:29.791470 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:29.791393 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2fbhq" Jul 10 20:39:32.601815 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:32.601748 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls\") pod \"dns-default-2fbhq\" (UID: \"2974e2ef-e853-4b53-8684-35405b1110fe\") " pod="openshift-dns/dns-default-2fbhq" Jul 10 20:39:32.604292 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:32.604263 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2974e2ef-e853-4b53-8684-35405b1110fe-metrics-tls\") pod \"dns-default-2fbhq\" (UID: \"2974e2ef-e853-4b53-8684-35405b1110fe\") " pod="openshift-dns/dns-default-2fbhq" Jul 10 20:39:32.794567 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:32.794534 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j2m8z\"" Jul 10 20:39:32.803197 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:32.803177 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2fbhq" Jul 10 20:39:32.941539 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:32.941509 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2fbhq"] Jul 10 20:39:32.944950 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:39:32.944924 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2974e2ef_e853_4b53_8684_35405b1110fe.slice/crio-7cd8655a96172fb0ec00c11df6701c131b7667644a36668b2ad540b4d33c5fe4 WatchSource:0}: Error finding container 7cd8655a96172fb0ec00c11df6701c131b7667644a36668b2ad540b4d33c5fe4: Status 404 returned error can't find the container with id 7cd8655a96172fb0ec00c11df6701c131b7667644a36668b2ad540b4d33c5fe4 Jul 10 20:39:33.805428 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:33.805389 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2fbhq" event={"ID":"2974e2ef-e853-4b53-8684-35405b1110fe","Type":"ContainerStarted","Data":"7cd8655a96172fb0ec00c11df6701c131b7667644a36668b2ad540b4d33c5fe4"} Jul 10 20:39:34.810018 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:34.809979 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2fbhq" event={"ID":"2974e2ef-e853-4b53-8684-35405b1110fe","Type":"ContainerStarted","Data":"c1258de8955b5e82601202084017ebcf6c7f5f650aa819b5bfc7556926c5e75f"} Jul 10 20:39:34.810411 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:34.810066 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2fbhq" event={"ID":"2974e2ef-e853-4b53-8684-35405b1110fe","Type":"ContainerStarted","Data":"6817ae9b66aa9e71d117ff0469d5759c9dec3d3416abd343caed3ef8078ee013"} Jul 10 20:39:34.810411 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:34.810090 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2fbhq" Jul 10 20:39:34.828148 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:34.828095 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2fbhq" podStartSLOduration=251.176659284 podStartE2EDuration="4m12.828080166s" totalImagesPullingTime="1.651420882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:35:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:39:34.826854474 +0000 UTC m=+287.605435859" watchObservedRunningTime="2026-07-10 20:39:34.828080166 +0000 UTC m=+287.606661560" Jul 10 20:39:36.078096 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.078064 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9gqvk"] Jul 10 20:39:36.080349 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.080318 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.082678 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.082659 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 20:39:36.082962 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.082948 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 20:39:36.083596 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.083575 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-99dbm\"" Jul 10 20:39:36.083596 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.083590 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 20:39:36.083758 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.083605 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 20:39:36.083758 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.083579 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 20:39:36.083758 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.083608 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 20:39:36.135625 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.135589 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f8317950-0804-43bf-82e3-e374099dc27b-root\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.135625 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.135626 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbfnc\" (UniqueName: \"kubernetes.io/projected/f8317950-0804-43bf-82e3-e374099dc27b-kube-api-access-sbfnc\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.135875 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.135645 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8317950-0804-43bf-82e3-e374099dc27b-sys\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.135875 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.135665 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f8317950-0804-43bf-82e3-e374099dc27b-node-exporter-accelerators-collector-config\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.135875 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.135683 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f8317950-0804-43bf-82e3-e374099dc27b-node-exporter-textfile\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.135875 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.135705 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8317950-0804-43bf-82e3-e374099dc27b-node-exporter-tls\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.135875 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.135719 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8317950-0804-43bf-82e3-e374099dc27b-metrics-client-ca\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.135875 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.135733 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8317950-0804-43bf-82e3-e374099dc27b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.135875 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.135842 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f8317950-0804-43bf-82e3-e374099dc27b-node-exporter-wtmp\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.236358 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.236317 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f8317950-0804-43bf-82e3-e374099dc27b-root\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.236358 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.236355 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbfnc\" (UniqueName: \"kubernetes.io/projected/f8317950-0804-43bf-82e3-e374099dc27b-kube-api-access-sbfnc\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.236641 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.236378 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8317950-0804-43bf-82e3-e374099dc27b-sys\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.236641 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.236425 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8317950-0804-43bf-82e3-e374099dc27b-sys\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.236641 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.236439 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f8317950-0804-43bf-82e3-e374099dc27b-root\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.236641 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.236491 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f8317950-0804-43bf-82e3-e374099dc27b-node-exporter-accelerators-collector-config\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.236641 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.236526 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f8317950-0804-43bf-82e3-e374099dc27b-node-exporter-textfile\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.236641 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.236564 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8317950-0804-43bf-82e3-e374099dc27b-node-exporter-tls\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.236641 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.236586 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8317950-0804-43bf-82e3-e374099dc27b-metrics-client-ca\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.236641 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.236611 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8317950-0804-43bf-82e3-e374099dc27b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.237007 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:39:36.236703 2530 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 10 20:39:36.237007 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:39:36.236782 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8317950-0804-43bf-82e3-e374099dc27b-node-exporter-tls podName:f8317950-0804-43bf-82e3-e374099dc27b nodeName:}" failed. No retries permitted until 2026-07-10 20:39:36.736759546 +0000 UTC m=+289.515340937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f8317950-0804-43bf-82e3-e374099dc27b-node-exporter-tls") pod "node-exporter-9gqvk" (UID: "f8317950-0804-43bf-82e3-e374099dc27b") : secret "node-exporter-tls" not found Jul 10 20:39:36.237007 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.236807 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f8317950-0804-43bf-82e3-e374099dc27b-node-exporter-wtmp\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.237007 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.236983 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f8317950-0804-43bf-82e3-e374099dc27b-node-exporter-textfile\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.237007 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.236990 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f8317950-0804-43bf-82e3-e374099dc27b-node-exporter-wtmp\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.237275 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.237156 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f8317950-0804-43bf-82e3-e374099dc27b-node-exporter-accelerators-collector-config\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.237330 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.237306 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8317950-0804-43bf-82e3-e374099dc27b-metrics-client-ca\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.239007 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.238986 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8317950-0804-43bf-82e3-e374099dc27b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.248992 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.248966 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbfnc\" (UniqueName: \"kubernetes.io/projected/f8317950-0804-43bf-82e3-e374099dc27b-kube-api-access-sbfnc\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.740479 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:36.740429 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8317950-0804-43bf-82e3-e374099dc27b-node-exporter-tls\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:36.740668 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:39:36.740578 2530 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 10 20:39:36.740668 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:39:36.740643 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8317950-0804-43bf-82e3-e374099dc27b-node-exporter-tls podName:f8317950-0804-43bf-82e3-e374099dc27b nodeName:}" failed. No retries permitted until 2026-07-10 20:39:37.740627973 +0000 UTC m=+290.519209350 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f8317950-0804-43bf-82e3-e374099dc27b-node-exporter-tls") pod "node-exporter-9gqvk" (UID: "f8317950-0804-43bf-82e3-e374099dc27b") : secret "node-exporter-tls" not found Jul 10 20:39:37.749089 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:37.749059 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8317950-0804-43bf-82e3-e374099dc27b-node-exporter-tls\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:37.751226 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:37.751207 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8317950-0804-43bf-82e3-e374099dc27b-node-exporter-tls\") pod \"node-exporter-9gqvk\" (UID: \"f8317950-0804-43bf-82e3-e374099dc27b\") " pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:37.869590 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:37.869552 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fbdd9bbdd-mxcqx"] Jul 10 20:39:37.889353 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:37.889326 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9gqvk" Jul 10 20:39:37.898881 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:39:37.898852 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8317950_0804_43bf_82e3_e374099dc27b.slice/crio-78db34cf51a578fe2d9d7421fc19955fd63f81ce3bcaafffe4598aa78c44a57d WatchSource:0}: Error finding container 78db34cf51a578fe2d9d7421fc19955fd63f81ce3bcaafffe4598aa78c44a57d: Status 404 returned error can't find the container with id 78db34cf51a578fe2d9d7421fc19955fd63f81ce3bcaafffe4598aa78c44a57d Jul 10 20:39:38.828644 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:38.828608 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9gqvk" event={"ID":"f8317950-0804-43bf-82e3-e374099dc27b","Type":"ContainerStarted","Data":"78db34cf51a578fe2d9d7421fc19955fd63f81ce3bcaafffe4598aa78c44a57d"} Jul 10 20:39:39.833364 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:39.833327 2530 generic.go:356] "Generic (PLEG): container finished" podID="f8317950-0804-43bf-82e3-e374099dc27b" containerID="c258a17681fa5b27fd51015b2670c0b4a228f324a77c2002b3f2d76d30d98700" exitCode=0 Jul 10 20:39:39.833875 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:39.833375 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9gqvk" event={"ID":"f8317950-0804-43bf-82e3-e374099dc27b","Type":"ContainerDied","Data":"c258a17681fa5b27fd51015b2670c0b4a228f324a77c2002b3f2d76d30d98700"} Jul 10 20:39:40.838987 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:40.838957 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9gqvk" event={"ID":"f8317950-0804-43bf-82e3-e374099dc27b","Type":"ContainerStarted","Data":"ba199a2ab5bff267bd1dbaa634db8e03646a2efd6b1a785d97c3d69fbd416ff2"} Jul 10 20:39:40.839384 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:40.839007 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9gqvk" event={"ID":"f8317950-0804-43bf-82e3-e374099dc27b","Type":"ContainerStarted","Data":"6269fb069131d9062966f19b5118f96338159a133645ce3629f5c979f789e706"} Jul 10 20:39:40.859262 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:40.859198 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9gqvk" podStartSLOduration=3.860549109 podStartE2EDuration="4.859182721s" totalImagesPullingTime="932.901631ms" totalInitContainerRuntime="65.731981ms" isStatefulPod=true podCreationTimestamp="2026-07-10 20:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:39:40.857446043 +0000 UTC m=+293.636027448" watchObservedRunningTime="2026-07-10 20:39:40.859182721 +0000 UTC m=+293.637764115" Jul 10 20:39:42.335913 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.335877 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 20:39:42.338447 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.338425 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.340842 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.340816 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 10 20:39:42.340967 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.340925 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 10 20:39:42.341075 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.341060 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mtznw\"" Jul 10 20:39:42.341075 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.341067 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 10 20:39:42.341381 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.341364 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 10 20:39:42.341731 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.341717 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 10 20:39:42.342427 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.342410 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 10 20:39:42.342479 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.342428 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 10 20:39:42.342681 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.342666 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 10 20:39:42.342930 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.342915 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-j1het276o2u6\"" Jul 10 20:39:42.343097 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.343083 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 10 20:39:42.343191 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.343174 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 10 20:39:42.343603 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.343591 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 10 20:39:42.343960 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.343946 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 10 20:39:42.344006 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.343949 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 10 20:39:42.349048 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.349030 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 10 20:39:42.349124 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.349031 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 10 20:39:42.356172 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.356149 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 20:39:42.486059 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.486025 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/207e13a4-5747-41b8-a458-025de50e8028-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.486059 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.486059 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.486286 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.486079 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.486286 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.486098 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-config\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.486286 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.486118 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207e13a4-5747-41b8-a458-025de50e8028-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.486286 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.486186 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.486286 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.486245 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bddhq\" (UniqueName: \"kubernetes.io/projected/207e13a4-5747-41b8-a458-025de50e8028-kube-api-access-bddhq\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.486286 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.486265 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-web-config\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.486462 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.486289 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/207e13a4-5747-41b8-a458-025de50e8028-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.486462 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.486313 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.486462 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.486330 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/207e13a4-5747-41b8-a458-025de50e8028-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.486462 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.486350 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.486462 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.486366 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.486462 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.486396 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/207e13a4-5747-41b8-a458-025de50e8028-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.486462 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.486420 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.486462 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.486442 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/207e13a4-5747-41b8-a458-025de50e8028-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.486695 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.486469 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/207e13a4-5747-41b8-a458-025de50e8028-config-out\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.486695 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.486484 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207e13a4-5747-41b8-a458-025de50e8028-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.486695 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.486556 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/207e13a4-5747-41b8-a458-025de50e8028-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.486695 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.486573 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207e13a4-5747-41b8-a458-025de50e8028-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.587490 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.587402 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/207e13a4-5747-41b8-a458-025de50e8028-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.587490 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.587443 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207e13a4-5747-41b8-a458-025de50e8028-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.587490 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.587477 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/207e13a4-5747-41b8-a458-025de50e8028-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.587716 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.587588 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.587716 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.587620 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.587716 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.587645 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-config\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.587716 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.587669 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207e13a4-5747-41b8-a458-025de50e8028-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.588090 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.588059 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.588210 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.588184 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/207e13a4-5747-41b8-a458-025de50e8028-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.588313 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.588295 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bddhq\" (UniqueName: \"kubernetes.io/projected/207e13a4-5747-41b8-a458-025de50e8028-kube-api-access-bddhq\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.588432 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.588416 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-web-config\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.588533 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.588510 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207e13a4-5747-41b8-a458-025de50e8028-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.588686 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.588361 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207e13a4-5747-41b8-a458-025de50e8028-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.588825 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.588535 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/207e13a4-5747-41b8-a458-025de50e8028-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.588975 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.588958 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.589083 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.589068 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/207e13a4-5747-41b8-a458-025de50e8028-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.589194 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.589170 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/207e13a4-5747-41b8-a458-025de50e8028-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.589333 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.589176 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.589449 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.589433 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.589551 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.589538 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/207e13a4-5747-41b8-a458-025de50e8028-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.589662 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.589644 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.589771 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.589756 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/207e13a4-5747-41b8-a458-025de50e8028-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.589905 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.589892 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/207e13a4-5747-41b8-a458-025de50e8028-config-out\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.590008 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.589994 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207e13a4-5747-41b8-a458-025de50e8028-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.590786 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.590755 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-config\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.590897 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.590793 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207e13a4-5747-41b8-a458-025de50e8028-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.590960 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.590937 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.591193 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.591170 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/207e13a4-5747-41b8-a458-025de50e8028-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.591275 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.591209 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-web-config\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.591328 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.591309 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.591727 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.591706 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/207e13a4-5747-41b8-a458-025de50e8028-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.591858 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.591726 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.591992 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.591859 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.592087 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.592023 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/207e13a4-5747-41b8-a458-025de50e8028-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.593250 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.593227 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/207e13a4-5747-41b8-a458-025de50e8028-config-out\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.593341 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.593281 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.593649 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.593632 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.594211 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.594192 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/207e13a4-5747-41b8-a458-025de50e8028-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.594598 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.594579 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/207e13a4-5747-41b8-a458-025de50e8028-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.598491 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.598471 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bddhq\" (UniqueName: \"kubernetes.io/projected/207e13a4-5747-41b8-a458-025de50e8028-kube-api-access-bddhq\") pod \"prometheus-k8s-0\" (UID: \"207e13a4-5747-41b8-a458-025de50e8028\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.648096 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.648053 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:42.785130 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.785031 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 20:39:42.787631 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:39:42.787602 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207e13a4_5747_41b8_a458_025de50e8028.slice/crio-26ce011c28b8b66fc8005bea48ac15daba843c713bd1a1223cbe3e528dda48ca WatchSource:0}: Error finding container 26ce011c28b8b66fc8005bea48ac15daba843c713bd1a1223cbe3e528dda48ca: Status 404 returned error can't find the container with id 26ce011c28b8b66fc8005bea48ac15daba843c713bd1a1223cbe3e528dda48ca Jul 10 20:39:42.846516 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:42.846432 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"207e13a4-5747-41b8-a458-025de50e8028","Type":"ContainerStarted","Data":"26ce011c28b8b66fc8005bea48ac15daba843c713bd1a1223cbe3e528dda48ca"} Jul 10 20:39:43.767961 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:39:43.767931 2530 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-dwvvg" podUID="93c55127-6ddc-4b15-844e-8b9585cc5d1e" Jul 10 20:39:43.851391 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:43.851361 2530 generic.go:356] "Generic (PLEG): container finished" podID="207e13a4-5747-41b8-a458-025de50e8028" containerID="001e3cfa48458b5d38bbec8cad1479784c3014ffa764d61334518afaae6dd1d9" exitCode=0 Jul 10 20:39:43.851551 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:43.851401 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"207e13a4-5747-41b8-a458-025de50e8028","Type":"ContainerDied","Data":"001e3cfa48458b5d38bbec8cad1479784c3014ffa764d61334518afaae6dd1d9"} Jul 10 20:39:44.815405 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:44.815367 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2fbhq" Jul 10 20:39:47.651613 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:47.651578 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n7kg6_a0fc144e-a4ca-4f83-9126-9366a781f0ad/ovn-acl-logging/0.log" Jul 10 20:39:47.652240 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:47.652222 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n7kg6_a0fc144e-a4ca-4f83-9126-9366a781f0ad/ovn-acl-logging/0.log" Jul 10 20:39:47.660093 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:47.660068 2530 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 20:39:47.872609 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:47.872453 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"207e13a4-5747-41b8-a458-025de50e8028","Type":"ContainerStarted","Data":"aae8ea1d089c1d77bfd68e0eacb8c177e7fdfceb5ae5a1d61767668e8bbee3e9"} Jul 10 20:39:47.872609 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:47.872525 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"207e13a4-5747-41b8-a458-025de50e8028","Type":"ContainerStarted","Data":"f3414aacc01cdb1a15f17abe3848025daae5cef3881641d2c2f03bf77532e1a0"} Jul 10 20:39:49.883304 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:49.883262 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"207e13a4-5747-41b8-a458-025de50e8028","Type":"ContainerStarted","Data":"07b955361d552e9e7d674a53665d56970e271fbe8b324b39f03b66d5e38fbba0"} Jul 10 20:39:49.883686 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:49.883345 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"207e13a4-5747-41b8-a458-025de50e8028","Type":"ContainerStarted","Data":"983977c7b4fa043bbeb7bad69577e26b3ae5f78958c153f9368ec2c6dbcc63b3"} Jul 10 20:39:49.883686 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:49.883386 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"207e13a4-5747-41b8-a458-025de50e8028","Type":"ContainerStarted","Data":"64e7590f2319b0bfc1385de93f7a0f85b637994ea604f54de8fc3e9510ac471e"} Jul 10 20:39:49.883686 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:49.883426 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"207e13a4-5747-41b8-a458-025de50e8028","Type":"ContainerStarted","Data":"4f70f36feea83f41963e6503a9b595983f969fb7148d966c46d24318940eef64"} Jul 10 20:39:49.910145 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:49.910088 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.564763661 podStartE2EDuration="7.91007201s" totalImagesPullingTime="6.328722174s" totalInitContainerRuntime="16.586175ms" isStatefulPod=false podCreationTimestamp="2026-07-10 20:39:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:39:49.908942421 +0000 UTC m=+302.687523815" watchObservedRunningTime="2026-07-10 20:39:49.91007201 +0000 UTC m=+302.688653404" Jul 10 20:39:52.648425 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:52.648384 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:39:55.767156 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:39:55.767114 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:40:02.889101 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:02.889040 2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7fbdd9bbdd-mxcqx" podUID="73da068e-4563-4406-a360-44b01cb07911" containerName="console" containerID="cri-o://010ab227998c8f4549f8b4bf2fef2985e94918b22e8d71a100bc63139daa5d70" gracePeriod=15 Jul 10 20:40:03.127916 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.127891 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbdd9bbdd-mxcqx" Jul 10 20:40:03.181300 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.181191 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73da068e-4563-4406-a360-44b01cb07911-console-config\") pod \"73da068e-4563-4406-a360-44b01cb07911\" (UID: \"73da068e-4563-4406-a360-44b01cb07911\") " Jul 10 20:40:03.181300 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.181258 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73da068e-4563-4406-a360-44b01cb07911-console-oauth-config\") pod \"73da068e-4563-4406-a360-44b01cb07911\" (UID: \"73da068e-4563-4406-a360-44b01cb07911\") " Jul 10 20:40:03.181533 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.181310 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fm88\" (UniqueName: \"kubernetes.io/projected/73da068e-4563-4406-a360-44b01cb07911-kube-api-access-6fm88\") pod \"73da068e-4563-4406-a360-44b01cb07911\" (UID: \"73da068e-4563-4406-a360-44b01cb07911\") " Jul 10 20:40:03.181533 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.181356 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73da068e-4563-4406-a360-44b01cb07911-oauth-serving-cert\") pod \"73da068e-4563-4406-a360-44b01cb07911\" (UID: \"73da068e-4563-4406-a360-44b01cb07911\") " Jul 10 20:40:03.181533 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.181381 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73da068e-4563-4406-a360-44b01cb07911-service-ca\") pod \"73da068e-4563-4406-a360-44b01cb07911\" (UID: \"73da068e-4563-4406-a360-44b01cb07911\") " Jul 10 20:40:03.181533 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.181407 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73da068e-4563-4406-a360-44b01cb07911-console-serving-cert\") pod \"73da068e-4563-4406-a360-44b01cb07911\" (UID: \"73da068e-4563-4406-a360-44b01cb07911\") " Jul 10 20:40:03.181721 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.181676 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73da068e-4563-4406-a360-44b01cb07911-console-config" (OuterVolumeSpecName: "console-config") pod "73da068e-4563-4406-a360-44b01cb07911" (UID: "73da068e-4563-4406-a360-44b01cb07911"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 20:40:03.181800 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.181769 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73da068e-4563-4406-a360-44b01cb07911-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "73da068e-4563-4406-a360-44b01cb07911" (UID: "73da068e-4563-4406-a360-44b01cb07911"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 20:40:03.181880 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.181793 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73da068e-4563-4406-a360-44b01cb07911-service-ca" (OuterVolumeSpecName: "service-ca") pod "73da068e-4563-4406-a360-44b01cb07911" (UID: "73da068e-4563-4406-a360-44b01cb07911"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 20:40:03.183505 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.183486 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73da068e-4563-4406-a360-44b01cb07911-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "73da068e-4563-4406-a360-44b01cb07911" (UID: "73da068e-4563-4406-a360-44b01cb07911"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:40:03.183580 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.183539 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73da068e-4563-4406-a360-44b01cb07911-kube-api-access-6fm88" (OuterVolumeSpecName: "kube-api-access-6fm88") pod "73da068e-4563-4406-a360-44b01cb07911" (UID: "73da068e-4563-4406-a360-44b01cb07911"). InnerVolumeSpecName "kube-api-access-6fm88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:40:03.183639 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.183609 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73da068e-4563-4406-a360-44b01cb07911-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "73da068e-4563-4406-a360-44b01cb07911" (UID: "73da068e-4563-4406-a360-44b01cb07911"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:40:03.281985 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.281931 2530 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73da068e-4563-4406-a360-44b01cb07911-console-config\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 20:40:03.281985 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.281976 2530 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73da068e-4563-4406-a360-44b01cb07911-console-oauth-config\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 20:40:03.281985 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.281994 2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fm88\" (UniqueName: \"kubernetes.io/projected/73da068e-4563-4406-a360-44b01cb07911-kube-api-access-6fm88\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 20:40:03.282219 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.282007 2530 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73da068e-4563-4406-a360-44b01cb07911-oauth-serving-cert\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 20:40:03.282219 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.282019 2530 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73da068e-4563-4406-a360-44b01cb07911-service-ca\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 20:40:03.282219 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.282030 2530 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73da068e-4563-4406-a360-44b01cb07911-console-serving-cert\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 20:40:03.931881 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.931822 2530 generic.go:356] "Generic (PLEG): container finished" podID="73da068e-4563-4406-a360-44b01cb07911" containerID="010ab227998c8f4549f8b4bf2fef2985e94918b22e8d71a100bc63139daa5d70" exitCode=0 Jul 10 20:40:03.932292 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.931889 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbdd9bbdd-mxcqx" Jul 10 20:40:03.932292 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.931907 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbdd9bbdd-mxcqx" event={"ID":"73da068e-4563-4406-a360-44b01cb07911","Type":"ContainerDied","Data":"010ab227998c8f4549f8b4bf2fef2985e94918b22e8d71a100bc63139daa5d70"} Jul 10 20:40:03.932292 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.931957 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbdd9bbdd-mxcqx" event={"ID":"73da068e-4563-4406-a360-44b01cb07911","Type":"ContainerDied","Data":"2d7589ea85aaafe0c159043e150614dab1c7b90a5dc15ebcb841507ccf0b4124"} Jul 10 20:40:03.932292 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.931989 2530 scope.go:117] "RemoveContainer" containerID="010ab227998c8f4549f8b4bf2fef2985e94918b22e8d71a100bc63139daa5d70" Jul 10 20:40:03.940163 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.940148 2530 scope.go:117] "RemoveContainer" containerID="010ab227998c8f4549f8b4bf2fef2985e94918b22e8d71a100bc63139daa5d70" Jul 10 20:40:03.940440 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:40:03.940422 2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"010ab227998c8f4549f8b4bf2fef2985e94918b22e8d71a100bc63139daa5d70\": container with ID starting with 010ab227998c8f4549f8b4bf2fef2985e94918b22e8d71a100bc63139daa5d70 not found: ID does not exist" containerID="010ab227998c8f4549f8b4bf2fef2985e94918b22e8d71a100bc63139daa5d70" Jul 10 20:40:03.940511 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.940448 2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"010ab227998c8f4549f8b4bf2fef2985e94918b22e8d71a100bc63139daa5d70"} err="failed to get container status \"010ab227998c8f4549f8b4bf2fef2985e94918b22e8d71a100bc63139daa5d70\": rpc error: code = NotFound desc = could not find container \"010ab227998c8f4549f8b4bf2fef2985e94918b22e8d71a100bc63139daa5d70\": container with ID starting with 010ab227998c8f4549f8b4bf2fef2985e94918b22e8d71a100bc63139daa5d70 not found: ID does not exist" Jul 10 20:40:03.947426 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.947392 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fbdd9bbdd-mxcqx"] Jul 10 20:40:03.951802 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:03.951781 2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fbdd9bbdd-mxcqx"] Jul 10 20:40:05.771590 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:05.771550 2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73da068e-4563-4406-a360-44b01cb07911" path="/var/lib/kubelet/pods/73da068e-4563-4406-a360-44b01cb07911/volumes" Jul 10 20:40:16.924594 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:16.924547 2530 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" podUID="9a547e11-0f61-4327-90bd-bbe2db1ceb34" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 10 20:40:26.924542 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:26.924504 2530 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" podUID="9a547e11-0f61-4327-90bd-bbe2db1ceb34" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 10 20:40:36.924399 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:36.924357 2530 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" podUID="9a547e11-0f61-4327-90bd-bbe2db1ceb34" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 10 20:40:36.924990 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:36.924437 2530 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" Jul 10 20:40:36.925118 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:36.925097 2530 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"82f60a2712c052a612d96eb79e410549ba5e7034a9c4db13d5808d90d3be0876"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 10 20:40:36.925180 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:36.925149 2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" podUID="9a547e11-0f61-4327-90bd-bbe2db1ceb34" containerName="service-proxy" containerID="cri-o://82f60a2712c052a612d96eb79e410549ba5e7034a9c4db13d5808d90d3be0876" gracePeriod=30 Jul 10 20:40:37.045342 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:37.045313 2530 generic.go:356] "Generic (PLEG): container finished" podID="9a547e11-0f61-4327-90bd-bbe2db1ceb34" containerID="82f60a2712c052a612d96eb79e410549ba5e7034a9c4db13d5808d90d3be0876" exitCode=2 Jul 10 20:40:37.045459 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:37.045380 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" event={"ID":"9a547e11-0f61-4327-90bd-bbe2db1ceb34","Type":"ContainerDied","Data":"82f60a2712c052a612d96eb79e410549ba5e7034a9c4db13d5808d90d3be0876"} Jul 10 20:40:38.050504 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:38.050468 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-846c4b5f54-vh4kt" event={"ID":"9a547e11-0f61-4327-90bd-bbe2db1ceb34","Type":"ContainerStarted","Data":"2a4819bc22932a990038cb8551f0eeef670a20f72254982eccedd8961377ca15"} Jul 10 20:40:42.648793 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:42.648700 2530 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:40:42.669657 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:42.669629 2530 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:40:43.085571 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:40:43.085487 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:41:00.652413 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:41:00.652363 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs\") pod \"network-metrics-daemon-dwvvg\" (UID: \"93c55127-6ddc-4b15-844e-8b9585cc5d1e\") " pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:41:00.654850 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:41:00.654810 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c55127-6ddc-4b15-844e-8b9585cc5d1e-metrics-certs\") pod \"network-metrics-daemon-dwvvg\" (UID: \"93c55127-6ddc-4b15-844e-8b9585cc5d1e\") " pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:41:00.869753 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:41:00.869716 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rvkjn\"" Jul 10 20:41:00.877771 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:41:00.877745 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwvvg" Jul 10 20:41:01.009197 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:41:01.009170 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dwvvg"] Jul 10 20:41:01.011935 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:41:01.011903 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c55127_6ddc_4b15_844e_8b9585cc5d1e.slice/crio-3b7cab87e8deaf4528734010d78899232118c553b50d4caed4bb56c6d1fb548a WatchSource:0}: Error finding container 3b7cab87e8deaf4528734010d78899232118c553b50d4caed4bb56c6d1fb548a: Status 404 returned error can't find the container with id 3b7cab87e8deaf4528734010d78899232118c553b50d4caed4bb56c6d1fb548a Jul 10 20:41:01.014340 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:41:01.014321 2530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 20:41:01.132415 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:41:01.132381 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dwvvg" event={"ID":"93c55127-6ddc-4b15-844e-8b9585cc5d1e","Type":"ContainerStarted","Data":"3b7cab87e8deaf4528734010d78899232118c553b50d4caed4bb56c6d1fb548a"} Jul 10 20:41:02.137604 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:41:02.137571 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dwvvg" event={"ID":"93c55127-6ddc-4b15-844e-8b9585cc5d1e","Type":"ContainerStarted","Data":"85a471c93c3c5f2e72b9e51f14d2ed6283cb5c18c61748bfb487711baef1f469"} Jul 10 20:41:02.137996 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:41:02.137630 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dwvvg" event={"ID":"93c55127-6ddc-4b15-844e-8b9585cc5d1e","Type":"ContainerStarted","Data":"798c8d047798f5aaef1c77bf752da5d86999df12ef13e6da6bcbda3927bd136a"} Jul 10 20:41:02.152126 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:41:02.152018 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dwvvg" podStartSLOduration=373.271136429 podStartE2EDuration="6m14.15199986s" totalImagesPullingTime="880.863431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:41:02.151546236 +0000 UTC m=+374.930127631" watchObservedRunningTime="2026-07-10 20:41:02.15199986 +0000 UTC m=+374.930581302" Jul 10 20:42:56.571759 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:56.571723 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c5b7cd584-mjlnp"] Jul 10 20:42:56.572258 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:56.571978 2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73da068e-4563-4406-a360-44b01cb07911" containerName="console" Jul 10 20:42:56.572258 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:56.571989 2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="73da068e-4563-4406-a360-44b01cb07911" containerName="console" Jul 10 20:42:56.572258 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:56.572040 2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="73da068e-4563-4406-a360-44b01cb07911" containerName="console" Jul 10 20:42:56.574870 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:56.574851 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c5b7cd584-mjlnp" Jul 10 20:42:56.577573 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:56.577544 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 10 20:42:56.579506 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:56.578551 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 20:42:56.579506 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:56.578793 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 10 20:42:56.579506 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:56.579033 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-c4tjt\"" Jul 10 20:42:56.579506 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:56.579320 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 20:42:56.579793 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:56.579542 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 10 20:42:56.585580 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:56.585556 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c5b7cd584-mjlnp"] Jul 10 20:42:56.635453 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:56.635415 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edf04234-9822-4e34-b7fb-905f0598fe27-cert\") pod \"lws-controller-manager-7c5b7cd584-mjlnp\" (UID: \"edf04234-9822-4e34-b7fb-905f0598fe27\") " pod="openshift-lws-operator/lws-controller-manager-7c5b7cd584-mjlnp" Jul 10 20:42:56.635453 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:56.635463 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/edf04234-9822-4e34-b7fb-905f0598fe27-manager-config\") pod \"lws-controller-manager-7c5b7cd584-mjlnp\" (UID: \"edf04234-9822-4e34-b7fb-905f0598fe27\") " pod="openshift-lws-operator/lws-controller-manager-7c5b7cd584-mjlnp" Jul 10 20:42:56.635679 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:56.635495 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/edf04234-9822-4e34-b7fb-905f0598fe27-metrics-cert\") pod \"lws-controller-manager-7c5b7cd584-mjlnp\" (UID: \"edf04234-9822-4e34-b7fb-905f0598fe27\") " pod="openshift-lws-operator/lws-controller-manager-7c5b7cd584-mjlnp" Jul 10 20:42:56.635679 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:56.635563 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r4sn\" (UniqueName: \"kubernetes.io/projected/edf04234-9822-4e34-b7fb-905f0598fe27-kube-api-access-8r4sn\") pod \"lws-controller-manager-7c5b7cd584-mjlnp\" (UID: \"edf04234-9822-4e34-b7fb-905f0598fe27\") " pod="openshift-lws-operator/lws-controller-manager-7c5b7cd584-mjlnp" Jul 10 20:42:56.736423 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:56.736374 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edf04234-9822-4e34-b7fb-905f0598fe27-cert\") pod \"lws-controller-manager-7c5b7cd584-mjlnp\" (UID: \"edf04234-9822-4e34-b7fb-905f0598fe27\") " pod="openshift-lws-operator/lws-controller-manager-7c5b7cd584-mjlnp" Jul 10 20:42:56.736604 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:56.736435 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/edf04234-9822-4e34-b7fb-905f0598fe27-manager-config\") pod \"lws-controller-manager-7c5b7cd584-mjlnp\" (UID: \"edf04234-9822-4e34-b7fb-905f0598fe27\") " pod="openshift-lws-operator/lws-controller-manager-7c5b7cd584-mjlnp" Jul 10 20:42:56.736604 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:56.736474 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/edf04234-9822-4e34-b7fb-905f0598fe27-metrics-cert\") pod \"lws-controller-manager-7c5b7cd584-mjlnp\" (UID: \"edf04234-9822-4e34-b7fb-905f0598fe27\") " pod="openshift-lws-operator/lws-controller-manager-7c5b7cd584-mjlnp" Jul 10 20:42:56.736604 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:56.736513 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r4sn\" (UniqueName: \"kubernetes.io/projected/edf04234-9822-4e34-b7fb-905f0598fe27-kube-api-access-8r4sn\") pod \"lws-controller-manager-7c5b7cd584-mjlnp\" (UID: \"edf04234-9822-4e34-b7fb-905f0598fe27\") " pod="openshift-lws-operator/lws-controller-manager-7c5b7cd584-mjlnp" Jul 10 20:42:56.737214 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:56.737184 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/edf04234-9822-4e34-b7fb-905f0598fe27-manager-config\") pod \"lws-controller-manager-7c5b7cd584-mjlnp\" (UID: \"edf04234-9822-4e34-b7fb-905f0598fe27\") " pod="openshift-lws-operator/lws-controller-manager-7c5b7cd584-mjlnp" Jul 10 20:42:56.738985 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:56.738963 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/edf04234-9822-4e34-b7fb-905f0598fe27-metrics-cert\") pod \"lws-controller-manager-7c5b7cd584-mjlnp\" (UID: \"edf04234-9822-4e34-b7fb-905f0598fe27\") " pod="openshift-lws-operator/lws-controller-manager-7c5b7cd584-mjlnp" Jul 10 20:42:56.739082 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:56.739037 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edf04234-9822-4e34-b7fb-905f0598fe27-cert\") pod \"lws-controller-manager-7c5b7cd584-mjlnp\" (UID: \"edf04234-9822-4e34-b7fb-905f0598fe27\") " pod="openshift-lws-operator/lws-controller-manager-7c5b7cd584-mjlnp" Jul 10 20:42:56.744485 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:56.744460 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r4sn\" (UniqueName: \"kubernetes.io/projected/edf04234-9822-4e34-b7fb-905f0598fe27-kube-api-access-8r4sn\") pod \"lws-controller-manager-7c5b7cd584-mjlnp\" (UID: \"edf04234-9822-4e34-b7fb-905f0598fe27\") " pod="openshift-lws-operator/lws-controller-manager-7c5b7cd584-mjlnp" Jul 10 20:42:56.886905 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:56.886862 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c5b7cd584-mjlnp" Jul 10 20:42:57.030212 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:57.030180 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c5b7cd584-mjlnp"] Jul 10 20:42:57.034221 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:42:57.034184 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf04234_9822_4e34_b7fb_905f0598fe27.slice/crio-28b97ec49d4e39c46fdcf8221765af0f0e8e3736b0ffc03efc5518eb7efa59e8 WatchSource:0}: Error finding container 28b97ec49d4e39c46fdcf8221765af0f0e8e3736b0ffc03efc5518eb7efa59e8: Status 404 returned error can't find the container with id 28b97ec49d4e39c46fdcf8221765af0f0e8e3736b0ffc03efc5518eb7efa59e8 Jul 10 20:42:57.532379 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:57.532303 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c5b7cd584-mjlnp" event={"ID":"edf04234-9822-4e34-b7fb-905f0598fe27","Type":"ContainerStarted","Data":"28b97ec49d4e39c46fdcf8221765af0f0e8e3736b0ffc03efc5518eb7efa59e8"} Jul 10 20:42:59.543961 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:59.543870 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c5b7cd584-mjlnp" event={"ID":"edf04234-9822-4e34-b7fb-905f0598fe27","Type":"ContainerStarted","Data":"90f9dce65853b77aa7681baa351ffc59c9800ce4f9d84594c3eb98981f2ef98f"} Jul 10 20:42:59.544321 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:59.543997 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7c5b7cd584-mjlnp" Jul 10 20:42:59.573673 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:42:59.573616 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7c5b7cd584-mjlnp" podStartSLOduration=1.454790745 podStartE2EDuration="3.573602294s" totalImagesPullingTime="2.118811549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:42:59.572197257 +0000 UTC m=+492.350778650" watchObservedRunningTime="2026-07-10 20:42:59.573602294 +0000 UTC m=+492.352183687" Jul 10 20:43:04.574632 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:04.574602 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk"] Jul 10 20:43:04.578070 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:04.578053 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk" Jul 10 20:43:04.581009 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:04.580985 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 20:43:04.581320 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:04.581303 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 20:43:04.581898 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:04.581880 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 20:43:04.582177 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:04.582157 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-t5qrt\"" Jul 10 20:43:04.582433 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:04.582418 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 20:43:04.608203 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:04.608174 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk"] Jul 10 20:43:04.699608 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:04.699552 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84rrm\" (UniqueName: \"kubernetes.io/projected/ed408813-2dcb-4d08-bec9-879a3018d2b8-kube-api-access-84rrm\") pod \"opendatahub-operator-controller-manager-d5798c7df-9t9rk\" (UID: \"ed408813-2dcb-4d08-bec9-879a3018d2b8\") " pod="opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk" Jul 10 20:43:04.699608 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:04.699611 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ed408813-2dcb-4d08-bec9-879a3018d2b8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d5798c7df-9t9rk\" (UID: \"ed408813-2dcb-4d08-bec9-879a3018d2b8\") " pod="opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk" Jul 10 20:43:04.699890 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:04.699635 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed408813-2dcb-4d08-bec9-879a3018d2b8-webhook-cert\") pod \"opendatahub-operator-controller-manager-d5798c7df-9t9rk\" (UID: \"ed408813-2dcb-4d08-bec9-879a3018d2b8\") " pod="opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk" Jul 10 20:43:04.699890 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:04.699670 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ed408813-2dcb-4d08-bec9-879a3018d2b8-manifests\") pod \"opendatahub-operator-controller-manager-d5798c7df-9t9rk\" (UID: \"ed408813-2dcb-4d08-bec9-879a3018d2b8\") " pod="opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk" Jul 10 20:43:04.699890 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:04.699730 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed408813-2dcb-4d08-bec9-879a3018d2b8-tmp\") pod \"opendatahub-operator-controller-manager-d5798c7df-9t9rk\" (UID: \"ed408813-2dcb-4d08-bec9-879a3018d2b8\") " pod="opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk" Jul 10 20:43:04.800498 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:04.800456 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84rrm\" (UniqueName: \"kubernetes.io/projected/ed408813-2dcb-4d08-bec9-879a3018d2b8-kube-api-access-84rrm\") pod \"opendatahub-operator-controller-manager-d5798c7df-9t9rk\" (UID: \"ed408813-2dcb-4d08-bec9-879a3018d2b8\") " pod="opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk" Jul 10 20:43:04.800498 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:04.800499 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ed408813-2dcb-4d08-bec9-879a3018d2b8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d5798c7df-9t9rk\" (UID: \"ed408813-2dcb-4d08-bec9-879a3018d2b8\") " pod="opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk" Jul 10 20:43:04.800714 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:04.800527 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed408813-2dcb-4d08-bec9-879a3018d2b8-webhook-cert\") pod \"opendatahub-operator-controller-manager-d5798c7df-9t9rk\" (UID: \"ed408813-2dcb-4d08-bec9-879a3018d2b8\") " pod="opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk" Jul 10 20:43:04.800714 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:04.800583 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ed408813-2dcb-4d08-bec9-879a3018d2b8-manifests\") pod \"opendatahub-operator-controller-manager-d5798c7df-9t9rk\" (UID: \"ed408813-2dcb-4d08-bec9-879a3018d2b8\") " pod="opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk" Jul 10 20:43:04.800714 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:04.800662 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed408813-2dcb-4d08-bec9-879a3018d2b8-tmp\") pod \"opendatahub-operator-controller-manager-d5798c7df-9t9rk\" (UID: \"ed408813-2dcb-4d08-bec9-879a3018d2b8\") " pod="opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk" Jul 10 20:43:04.801040 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:04.801019 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ed408813-2dcb-4d08-bec9-879a3018d2b8-manifests\") pod \"opendatahub-operator-controller-manager-d5798c7df-9t9rk\" (UID: \"ed408813-2dcb-4d08-bec9-879a3018d2b8\") " pod="opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk" Jul 10 20:43:04.802819 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:04.802797 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed408813-2dcb-4d08-bec9-879a3018d2b8-tmp\") pod \"opendatahub-operator-controller-manager-d5798c7df-9t9rk\" (UID: \"ed408813-2dcb-4d08-bec9-879a3018d2b8\") " pod="opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk" Jul 10 20:43:04.803064 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:04.803045 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ed408813-2dcb-4d08-bec9-879a3018d2b8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d5798c7df-9t9rk\" (UID: \"ed408813-2dcb-4d08-bec9-879a3018d2b8\") " pod="opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk" Jul 10 20:43:04.803157 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:04.803139 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed408813-2dcb-4d08-bec9-879a3018d2b8-webhook-cert\") pod \"opendatahub-operator-controller-manager-d5798c7df-9t9rk\" (UID: \"ed408813-2dcb-4d08-bec9-879a3018d2b8\") " pod="opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk" Jul 10 20:43:04.807671 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:04.807645 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84rrm\" (UniqueName: \"kubernetes.io/projected/ed408813-2dcb-4d08-bec9-879a3018d2b8-kube-api-access-84rrm\") pod \"opendatahub-operator-controller-manager-d5798c7df-9t9rk\" (UID: \"ed408813-2dcb-4d08-bec9-879a3018d2b8\") " pod="opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk" Jul 10 20:43:04.887938 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:04.887912 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk" Jul 10 20:43:05.016110 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:05.016084 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk"] Jul 10 20:43:05.020734 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:43:05.020703 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded408813_2dcb_4d08_bec9_879a3018d2b8.slice/crio-05fbf06dbc46b0ca370732de284edc1232d731163653cf4d7e722a7f3187c8e7 WatchSource:0}: Error finding container 05fbf06dbc46b0ca370732de284edc1232d731163653cf4d7e722a7f3187c8e7: Status 404 returned error can't find the container with id 05fbf06dbc46b0ca370732de284edc1232d731163653cf4d7e722a7f3187c8e7 Jul 10 20:43:05.565083 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:05.565045 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk" event={"ID":"ed408813-2dcb-4d08-bec9-879a3018d2b8","Type":"ContainerStarted","Data":"05fbf06dbc46b0ca370732de284edc1232d731163653cf4d7e722a7f3187c8e7"} Jul 10 20:43:08.579215 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:08.579183 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk" event={"ID":"ed408813-2dcb-4d08-bec9-879a3018d2b8","Type":"ContainerStarted","Data":"7316c17be7acc62e3e2ef85171b3c4ac8775a9d8c7c9576bfca06b1f913a22cd"} Jul 10 20:43:09.584855 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:09.584803 2530 generic.go:356] "Generic (PLEG): container finished" podID="ed408813-2dcb-4d08-bec9-879a3018d2b8" containerID="7316c17be7acc62e3e2ef85171b3c4ac8775a9d8c7c9576bfca06b1f913a22cd" exitCode=0 Jul 10 20:43:09.585311 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:09.584900 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk" event={"ID":"ed408813-2dcb-4d08-bec9-879a3018d2b8","Type":"ContainerDied","Data":"7316c17be7acc62e3e2ef85171b3c4ac8775a9d8c7c9576bfca06b1f913a22cd"} Jul 10 20:43:10.549243 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:10.549213 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7c5b7cd584-mjlnp" Jul 10 20:43:10.590874 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:10.590796 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk" event={"ID":"ed408813-2dcb-4d08-bec9-879a3018d2b8","Type":"ContainerStarted","Data":"fb90487e9d1b5d82f181bb7154e17d8f39576f40b1f7c51d27d5e69e510d4566"} Jul 10 20:43:10.591271 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:10.590956 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk" Jul 10 20:43:10.611603 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:10.611549 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk" podStartSLOduration=2.1791652360000002 podStartE2EDuration="6.611532546s" totalImagesPullingTime="3.408781831s" totalInitContainerRuntime="1.023585479s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:43:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:43:10.610109151 +0000 UTC m=+503.388690545" watchObservedRunningTime="2026-07-10 20:43:10.611532546 +0000 UTC m=+503.390113939" Jul 10 20:43:21.597014 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:21.596977 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d5798c7df-9t9rk" Jul 10 20:43:32.451483 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.451450 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt"] Jul 10 20:43:32.467313 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.467281 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" Jul 10 20:43:32.468512 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.468456 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt"] Jul 10 20:43:32.469657 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.469635 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h46w4\"" Jul 10 20:43:32.469784 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.469694 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 20:43:32.469784 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.469695 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 20:43:32.469784 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.469633 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 10 20:43:32.469784 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.469701 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 10 20:43:32.469784 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.469713 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 20:43:32.470046 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.470028 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 10 20:43:32.520222 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.520180 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b6361b68-16e0-4d13-8439-095977bfcedc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-9hnkt\" (UID: \"b6361b68-16e0-4d13-8439-095977bfcedc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" Jul 10 20:43:32.520414 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.520231 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dm7t\" (UniqueName: \"kubernetes.io/projected/b6361b68-16e0-4d13-8439-095977bfcedc-kube-api-access-6dm7t\") pod \"istiod-openshift-gateway-94bb8fbfd-9hnkt\" (UID: \"b6361b68-16e0-4d13-8439-095977bfcedc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" Jul 10 20:43:32.520414 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.520250 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6361b68-16e0-4d13-8439-095977bfcedc-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-9hnkt\" (UID: \"b6361b68-16e0-4d13-8439-095977bfcedc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" Jul 10 20:43:32.520414 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.520312 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b6361b68-16e0-4d13-8439-095977bfcedc-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-9hnkt\" (UID: \"b6361b68-16e0-4d13-8439-095977bfcedc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" Jul 10 20:43:32.520414 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.520351 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6361b68-16e0-4d13-8439-095977bfcedc-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-9hnkt\" (UID: \"b6361b68-16e0-4d13-8439-095977bfcedc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" Jul 10 20:43:32.520414 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.520375 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b6361b68-16e0-4d13-8439-095977bfcedc-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-9hnkt\" (UID: \"b6361b68-16e0-4d13-8439-095977bfcedc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" Jul 10 20:43:32.520414 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.520404 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b6361b68-16e0-4d13-8439-095977bfcedc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-9hnkt\" (UID: \"b6361b68-16e0-4d13-8439-095977bfcedc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" Jul 10 20:43:32.621467 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.621416 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b6361b68-16e0-4d13-8439-095977bfcedc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-9hnkt\" (UID: \"b6361b68-16e0-4d13-8439-095977bfcedc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" Jul 10 20:43:32.621467 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.621471 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b6361b68-16e0-4d13-8439-095977bfcedc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-9hnkt\" (UID: \"b6361b68-16e0-4d13-8439-095977bfcedc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" Jul 10 20:43:32.621670 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.621513 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dm7t\" (UniqueName: \"kubernetes.io/projected/b6361b68-16e0-4d13-8439-095977bfcedc-kube-api-access-6dm7t\") pod \"istiod-openshift-gateway-94bb8fbfd-9hnkt\" (UID: \"b6361b68-16e0-4d13-8439-095977bfcedc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" Jul 10 20:43:32.621670 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.621532 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6361b68-16e0-4d13-8439-095977bfcedc-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-9hnkt\" (UID: \"b6361b68-16e0-4d13-8439-095977bfcedc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" Jul 10 20:43:32.621745 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.621670 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b6361b68-16e0-4d13-8439-095977bfcedc-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-9hnkt\" (UID: \"b6361b68-16e0-4d13-8439-095977bfcedc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" Jul 10 20:43:32.621745 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.621721 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6361b68-16e0-4d13-8439-095977bfcedc-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-9hnkt\" (UID: \"b6361b68-16e0-4d13-8439-095977bfcedc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" Jul 10 20:43:32.621861 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.621756 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b6361b68-16e0-4d13-8439-095977bfcedc-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-9hnkt\" (UID: \"b6361b68-16e0-4d13-8439-095977bfcedc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" Jul 10 20:43:32.622173 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.622145 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b6361b68-16e0-4d13-8439-095977bfcedc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-9hnkt\" (UID: \"b6361b68-16e0-4d13-8439-095977bfcedc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" Jul 10 20:43:32.624025 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.623995 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b6361b68-16e0-4d13-8439-095977bfcedc-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-9hnkt\" (UID: \"b6361b68-16e0-4d13-8439-095977bfcedc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" Jul 10 20:43:32.624151 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.624074 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b6361b68-16e0-4d13-8439-095977bfcedc-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-9hnkt\" (UID: \"b6361b68-16e0-4d13-8439-095977bfcedc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" Jul 10 20:43:32.624151 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.624084 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6361b68-16e0-4d13-8439-095977bfcedc-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-9hnkt\" (UID: \"b6361b68-16e0-4d13-8439-095977bfcedc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" Jul 10 20:43:32.624240 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.624154 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b6361b68-16e0-4d13-8439-095977bfcedc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-9hnkt\" (UID: \"b6361b68-16e0-4d13-8439-095977bfcedc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" Jul 10 20:43:32.636341 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.636314 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dm7t\" (UniqueName: \"kubernetes.io/projected/b6361b68-16e0-4d13-8439-095977bfcedc-kube-api-access-6dm7t\") pod \"istiod-openshift-gateway-94bb8fbfd-9hnkt\" (UID: \"b6361b68-16e0-4d13-8439-095977bfcedc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" Jul 10 20:43:32.655585 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.655556 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6361b68-16e0-4d13-8439-095977bfcedc-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-9hnkt\" (UID: \"b6361b68-16e0-4d13-8439-095977bfcedc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" Jul 10 20:43:32.777769 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.777686 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" Jul 10 20:43:32.907661 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:32.907639 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt"] Jul 10 20:43:32.909962 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:43:32.909934 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6361b68_16e0_4d13_8439_095977bfcedc.slice/crio-2d591ae3f2a89238ee58c10a29387af5078d82a04b6bbdbe66428ef1e2ae2299 WatchSource:0}: Error finding container 2d591ae3f2a89238ee58c10a29387af5078d82a04b6bbdbe66428ef1e2ae2299: Status 404 returned error can't find the container with id 2d591ae3f2a89238ee58c10a29387af5078d82a04b6bbdbe66428ef1e2ae2299 Jul 10 20:43:33.675192 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:33.675151 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" event={"ID":"b6361b68-16e0-4d13-8439-095977bfcedc","Type":"ContainerStarted","Data":"2d591ae3f2a89238ee58c10a29387af5078d82a04b6bbdbe66428ef1e2ae2299"} Jul 10 20:43:36.285390 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:36.285355 2530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 20:43:36.285650 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:36.285434 2530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 20:43:36.688458 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:36.688423 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" event={"ID":"b6361b68-16e0-4d13-8439-095977bfcedc","Type":"ContainerStarted","Data":"57b9b54c090be3b7ccf4cb8290ef5db3e138c88d2bef34d6e6d6bac248d83722"} Jul 10 20:43:36.688662 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:36.688526 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" Jul 10 20:43:37.694655 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:37.694627 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" Jul 10 20:43:37.717678 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:43:37.717626 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-9hnkt" podStartSLOduration=2.344407935 podStartE2EDuration="5.717609011s" totalImagesPullingTime="3.373201076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:43:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:43:36.714129329 +0000 UTC m=+529.492710722" watchObservedRunningTime="2026-07-10 20:43:37.717609011 +0000 UTC m=+530.496190404" Jul 10 20:44:47.675539 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:44:47.675510 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n7kg6_a0fc144e-a4ca-4f83-9126-9366a781f0ad/ovn-acl-logging/0.log" Jul 10 20:44:47.677043 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:44:47.677021 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n7kg6_a0fc144e-a4ca-4f83-9126-9366a781f0ad/ovn-acl-logging/0.log" Jul 10 20:46:12.553107 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:12.553077 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-22s4q"] Jul 10 20:46:12.556225 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:12.556209 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-22s4q" Jul 10 20:46:12.560753 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:12.560726 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 20:46:12.560919 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:12.560768 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 10 20:46:12.560919 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:12.560771 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vbv2p\"" Jul 10 20:46:12.561434 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:12.561417 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 20:46:12.577131 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:12.577096 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-22s4q"] Jul 10 20:46:12.595368 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:12.595336 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z9pg\" (UniqueName: \"kubernetes.io/projected/b680428e-d1a9-4c83-acf2-082ee5a80585-kube-api-access-8z9pg\") pod \"limitador-limitador-64c94fd64f-22s4q\" (UID: \"b680428e-d1a9-4c83-acf2-082ee5a80585\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-22s4q" Jul 10 20:46:12.595510 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:12.595379 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b680428e-d1a9-4c83-acf2-082ee5a80585-config-file\") pod \"limitador-limitador-64c94fd64f-22s4q\" (UID: \"b680428e-d1a9-4c83-acf2-082ee5a80585\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-22s4q" Jul 10 20:46:12.638874 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:12.638824 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-22s4q"] Jul 10 20:46:12.696324 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:12.696283 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z9pg\" (UniqueName: \"kubernetes.io/projected/b680428e-d1a9-4c83-acf2-082ee5a80585-kube-api-access-8z9pg\") pod \"limitador-limitador-64c94fd64f-22s4q\" (UID: \"b680428e-d1a9-4c83-acf2-082ee5a80585\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-22s4q" Jul 10 20:46:12.696324 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:12.696325 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b680428e-d1a9-4c83-acf2-082ee5a80585-config-file\") pod \"limitador-limitador-64c94fd64f-22s4q\" (UID: \"b680428e-d1a9-4c83-acf2-082ee5a80585\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-22s4q" Jul 10 20:46:12.696927 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:12.696910 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b680428e-d1a9-4c83-acf2-082ee5a80585-config-file\") pod \"limitador-limitador-64c94fd64f-22s4q\" (UID: \"b680428e-d1a9-4c83-acf2-082ee5a80585\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-22s4q" Jul 10 20:46:12.705744 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:12.705720 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z9pg\" (UniqueName: \"kubernetes.io/projected/b680428e-d1a9-4c83-acf2-082ee5a80585-kube-api-access-8z9pg\") pod \"limitador-limitador-64c94fd64f-22s4q\" (UID: \"b680428e-d1a9-4c83-acf2-082ee5a80585\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-22s4q" Jul 10 20:46:12.866865 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:12.866757 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-22s4q" Jul 10 20:46:12.998191 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:12.998154 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-22s4q"] Jul 10 20:46:13.001289 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:46:13.001253 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb680428e_d1a9_4c83_acf2_082ee5a80585.slice/crio-1da05791f28b06573ac1ea9074a3035e37787b14b7dc561a5ec0475e278d8e1e WatchSource:0}: Error finding container 1da05791f28b06573ac1ea9074a3035e37787b14b7dc561a5ec0475e278d8e1e: Status 404 returned error can't find the container with id 1da05791f28b06573ac1ea9074a3035e37787b14b7dc561a5ec0475e278d8e1e Jul 10 20:46:13.003236 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:13.003221 2530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 20:46:13.269752 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:13.269721 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-22s4q" event={"ID":"b680428e-d1a9-4c83-acf2-082ee5a80585","Type":"ContainerStarted","Data":"1da05791f28b06573ac1ea9074a3035e37787b14b7dc561a5ec0475e278d8e1e"} Jul 10 20:46:16.283296 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:16.283265 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-22s4q" event={"ID":"b680428e-d1a9-4c83-acf2-082ee5a80585","Type":"ContainerStarted","Data":"3e4d3a09a081b16c55ebec11cf67f43887fa6e528f708e43a9a283abf94efeaa"} Jul 10 20:46:16.283702 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:16.283391 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-22s4q" Jul 10 20:46:27.287730 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:27.287693 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-22s4q" Jul 10 20:46:27.308675 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:27.308617 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-22s4q" podStartSLOduration=12.516010684 podStartE2EDuration="15.308598998s" totalImagesPullingTime="2.792588314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:46:16.320853983 +0000 UTC m=+689.099435374" watchObservedRunningTime="2026-07-10 20:46:27.308598998 +0000 UTC m=+700.087180392" Jul 10 20:46:27.759394 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:27.759358 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-22s4q"] Jul 10 20:46:27.759584 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:27.759538 2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-22s4q" podUID="b680428e-d1a9-4c83-acf2-082ee5a80585" containerName="limitador" containerID="cri-o://3e4d3a09a081b16c55ebec11cf67f43887fa6e528f708e43a9a283abf94efeaa" gracePeriod=30 Jul 10 20:46:28.309821 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:28.309796 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-22s4q" Jul 10 20:46:28.327927 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:28.327826 2530 generic.go:356] "Generic (PLEG): container finished" podID="b680428e-d1a9-4c83-acf2-082ee5a80585" containerID="3e4d3a09a081b16c55ebec11cf67f43887fa6e528f708e43a9a283abf94efeaa" exitCode=0 Jul 10 20:46:28.327927 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:28.327884 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-22s4q" Jul 10 20:46:28.327927 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:28.327895 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-22s4q" event={"ID":"b680428e-d1a9-4c83-acf2-082ee5a80585","Type":"ContainerDied","Data":"3e4d3a09a081b16c55ebec11cf67f43887fa6e528f708e43a9a283abf94efeaa"} Jul 10 20:46:28.328140 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:28.327943 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-22s4q" event={"ID":"b680428e-d1a9-4c83-acf2-082ee5a80585","Type":"ContainerDied","Data":"1da05791f28b06573ac1ea9074a3035e37787b14b7dc561a5ec0475e278d8e1e"} Jul 10 20:46:28.328140 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:28.327967 2530 scope.go:117] "RemoveContainer" containerID="3e4d3a09a081b16c55ebec11cf67f43887fa6e528f708e43a9a283abf94efeaa" Jul 10 20:46:28.336323 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:28.336214 2530 scope.go:117] "RemoveContainer" containerID="3e4d3a09a081b16c55ebec11cf67f43887fa6e528f708e43a9a283abf94efeaa" Jul 10 20:46:28.336715 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:46:28.336689 2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e4d3a09a081b16c55ebec11cf67f43887fa6e528f708e43a9a283abf94efeaa\": container with ID starting with 3e4d3a09a081b16c55ebec11cf67f43887fa6e528f708e43a9a283abf94efeaa not found: ID does not exist" containerID="3e4d3a09a081b16c55ebec11cf67f43887fa6e528f708e43a9a283abf94efeaa" Jul 10 20:46:28.336812 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:28.336726 2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4d3a09a081b16c55ebec11cf67f43887fa6e528f708e43a9a283abf94efeaa"} err="failed to get container status \"3e4d3a09a081b16c55ebec11cf67f43887fa6e528f708e43a9a283abf94efeaa\": rpc error: code = NotFound desc = could not find container \"3e4d3a09a081b16c55ebec11cf67f43887fa6e528f708e43a9a283abf94efeaa\": container with ID starting with 3e4d3a09a081b16c55ebec11cf67f43887fa6e528f708e43a9a283abf94efeaa not found: ID does not exist" Jul 10 20:46:28.431128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:28.431086 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b680428e-d1a9-4c83-acf2-082ee5a80585-config-file\") pod \"b680428e-d1a9-4c83-acf2-082ee5a80585\" (UID: \"b680428e-d1a9-4c83-acf2-082ee5a80585\") " Jul 10 20:46:28.431333 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:28.431145 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z9pg\" (UniqueName: \"kubernetes.io/projected/b680428e-d1a9-4c83-acf2-082ee5a80585-kube-api-access-8z9pg\") pod \"b680428e-d1a9-4c83-acf2-082ee5a80585\" (UID: \"b680428e-d1a9-4c83-acf2-082ee5a80585\") " Jul 10 20:46:28.431484 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:28.431457 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b680428e-d1a9-4c83-acf2-082ee5a80585-config-file" (OuterVolumeSpecName: "config-file") pod "b680428e-d1a9-4c83-acf2-082ee5a80585" (UID: "b680428e-d1a9-4c83-acf2-082ee5a80585"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 20:46:28.433348 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:28.433328 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b680428e-d1a9-4c83-acf2-082ee5a80585-kube-api-access-8z9pg" (OuterVolumeSpecName: "kube-api-access-8z9pg") pod "b680428e-d1a9-4c83-acf2-082ee5a80585" (UID: "b680428e-d1a9-4c83-acf2-082ee5a80585"). InnerVolumeSpecName "kube-api-access-8z9pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:46:28.531969 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:28.531929 2530 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b680428e-d1a9-4c83-acf2-082ee5a80585-config-file\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 20:46:28.531969 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:28.531960 2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z9pg\" (UniqueName: \"kubernetes.io/projected/b680428e-d1a9-4c83-acf2-082ee5a80585-kube-api-access-8z9pg\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 20:46:28.651222 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:28.651185 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-22s4q"] Jul 10 20:46:28.656182 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:28.656154 2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-22s4q"] Jul 10 20:46:29.773815 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:29.773776 2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b680428e-d1a9-4c83-acf2-082ee5a80585" path="/var/lib/kubelet/pods/b680428e-d1a9-4c83-acf2-082ee5a80585/volumes" Jul 10 20:46:34.181655 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:34.181615 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rjzz6"] Jul 10 20:46:34.182117 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:34.182044 2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b680428e-d1a9-4c83-acf2-082ee5a80585" containerName="limitador" Jul 10 20:46:34.182117 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:34.182061 2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="b680428e-d1a9-4c83-acf2-082ee5a80585" containerName="limitador" Jul 10 20:46:34.182188 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:34.182130 2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="b680428e-d1a9-4c83-acf2-082ee5a80585" containerName="limitador" Jul 10 20:46:34.188071 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:34.188050 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjzz6" Jul 10 20:46:34.190641 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:34.190613 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 10 20:46:34.190641 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:34.190613 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-c8kdg\"" Jul 10 20:46:34.191363 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:34.191343 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 20:46:34.191363 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:34.191358 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 20:46:34.196796 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:34.196773 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rjzz6"] Jul 10 20:46:34.278244 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:34.278195 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5afc7bda-a2d6-4dc4-a33d-10792e5a347a-data\") pod \"postgres-69fb4594cf-rjzz6\" (UID: \"5afc7bda-a2d6-4dc4-a33d-10792e5a347a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjzz6" Jul 10 20:46:34.278432 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:34.278259 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4cvx\" (UniqueName: \"kubernetes.io/projected/5afc7bda-a2d6-4dc4-a33d-10792e5a347a-kube-api-access-w4cvx\") pod \"postgres-69fb4594cf-rjzz6\" (UID: \"5afc7bda-a2d6-4dc4-a33d-10792e5a347a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjzz6" Jul 10 20:46:34.379140 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:34.379105 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5afc7bda-a2d6-4dc4-a33d-10792e5a347a-data\") pod \"postgres-69fb4594cf-rjzz6\" (UID: \"5afc7bda-a2d6-4dc4-a33d-10792e5a347a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjzz6" Jul 10 20:46:34.379301 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:34.379153 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4cvx\" (UniqueName: \"kubernetes.io/projected/5afc7bda-a2d6-4dc4-a33d-10792e5a347a-kube-api-access-w4cvx\") pod \"postgres-69fb4594cf-rjzz6\" (UID: \"5afc7bda-a2d6-4dc4-a33d-10792e5a347a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjzz6" Jul 10 20:46:34.379499 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:34.379476 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5afc7bda-a2d6-4dc4-a33d-10792e5a347a-data\") pod \"postgres-69fb4594cf-rjzz6\" (UID: \"5afc7bda-a2d6-4dc4-a33d-10792e5a347a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjzz6" Jul 10 20:46:34.388409 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:34.388380 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4cvx\" (UniqueName: \"kubernetes.io/projected/5afc7bda-a2d6-4dc4-a33d-10792e5a347a-kube-api-access-w4cvx\") pod \"postgres-69fb4594cf-rjzz6\" (UID: \"5afc7bda-a2d6-4dc4-a33d-10792e5a347a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjzz6" Jul 10 20:46:34.498729 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:34.498639 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjzz6" Jul 10 20:46:34.640566 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:34.640424 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rjzz6"] Jul 10 20:46:34.642753 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:46:34.642719 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5afc7bda_a2d6_4dc4_a33d_10792e5a347a.slice/crio-52f44537a1e2727a306d5a5f026a496b7d6028bcabafebbb5c52f3897aaabedc WatchSource:0}: Error finding container 52f44537a1e2727a306d5a5f026a496b7d6028bcabafebbb5c52f3897aaabedc: Status 404 returned error can't find the container with id 52f44537a1e2727a306d5a5f026a496b7d6028bcabafebbb5c52f3897aaabedc Jul 10 20:46:35.359793 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:35.359760 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjzz6" event={"ID":"5afc7bda-a2d6-4dc4-a33d-10792e5a347a","Type":"ContainerStarted","Data":"52f44537a1e2727a306d5a5f026a496b7d6028bcabafebbb5c52f3897aaabedc"} Jul 10 20:46:40.382056 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:40.382019 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjzz6" event={"ID":"5afc7bda-a2d6-4dc4-a33d-10792e5a347a","Type":"ContainerStarted","Data":"36a62acd4db35548b0622a80c9bc5838a5262cec845805148b8d8c44a0978472"} Jul 10 20:46:40.382485 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:40.382152 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjzz6" Jul 10 20:46:40.402624 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:40.402573 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjzz6" podStartSLOduration=1.234442536 podStartE2EDuration="6.402556828s" totalImagesPullingTime="5.168114292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:46:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:46:40.40138695 +0000 UTC m=+713.179968344" watchObservedRunningTime="2026-07-10 20:46:40.402556828 +0000 UTC m=+713.181138216" Jul 10 20:46:46.415397 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:46.415365 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjzz6" Jul 10 20:46:53.253168 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:53.253129 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-686b46dd7f-4d7pv"] Jul 10 20:46:53.257720 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:53.257696 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-686b46dd7f-4d7pv" Jul 10 20:46:53.260506 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:53.260479 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 20:46:53.260651 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:53.260631 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pd7v2\"" Jul 10 20:46:53.261593 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:53.261573 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 20:46:53.267378 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:53.267355 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-686b46dd7f-4d7pv"] Jul 10 20:46:53.330130 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:53.330097 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f72372b4-db0c-4a0a-848a-14860e3bd9ff-webhook-cert\") pod \"maas-controller-686b46dd7f-4d7pv\" (UID: \"f72372b4-db0c-4a0a-848a-14860e3bd9ff\") " pod="opendatahub/maas-controller-686b46dd7f-4d7pv" Jul 10 20:46:53.330130 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:53.330131 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdffn\" (UniqueName: \"kubernetes.io/projected/f72372b4-db0c-4a0a-848a-14860e3bd9ff-kube-api-access-gdffn\") pod \"maas-controller-686b46dd7f-4d7pv\" (UID: \"f72372b4-db0c-4a0a-848a-14860e3bd9ff\") " pod="opendatahub/maas-controller-686b46dd7f-4d7pv" Jul 10 20:46:53.430772 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:53.430740 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f72372b4-db0c-4a0a-848a-14860e3bd9ff-webhook-cert\") pod \"maas-controller-686b46dd7f-4d7pv\" (UID: \"f72372b4-db0c-4a0a-848a-14860e3bd9ff\") " pod="opendatahub/maas-controller-686b46dd7f-4d7pv" Jul 10 20:46:53.430973 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:53.430780 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdffn\" (UniqueName: \"kubernetes.io/projected/f72372b4-db0c-4a0a-848a-14860e3bd9ff-kube-api-access-gdffn\") pod \"maas-controller-686b46dd7f-4d7pv\" (UID: \"f72372b4-db0c-4a0a-848a-14860e3bd9ff\") " pod="opendatahub/maas-controller-686b46dd7f-4d7pv" Jul 10 20:46:53.430973 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:46:53.430906 2530 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 10 20:46:53.431054 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:46:53.430979 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f72372b4-db0c-4a0a-848a-14860e3bd9ff-webhook-cert podName:f72372b4-db0c-4a0a-848a-14860e3bd9ff nodeName:}" failed. No retries permitted until 2026-07-10 20:46:53.930961751 +0000 UTC m=+726.709543142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/f72372b4-db0c-4a0a-848a-14860e3bd9ff-webhook-cert") pod "maas-controller-686b46dd7f-4d7pv" (UID: "f72372b4-db0c-4a0a-848a-14860e3bd9ff") : secret "maas-controller-webhook-cert" not found Jul 10 20:46:53.441163 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:53.441126 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6bc6856dbb-ftct4"] Jul 10 20:46:53.442793 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:53.442763 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdffn\" (UniqueName: \"kubernetes.io/projected/f72372b4-db0c-4a0a-848a-14860e3bd9ff-kube-api-access-gdffn\") pod \"maas-controller-686b46dd7f-4d7pv\" (UID: \"f72372b4-db0c-4a0a-848a-14860e3bd9ff\") " pod="opendatahub/maas-controller-686b46dd7f-4d7pv" Jul 10 20:46:53.444187 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:53.444167 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bc6856dbb-ftct4" Jul 10 20:46:53.454506 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:53.454476 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bc6856dbb-ftct4"] Jul 10 20:46:53.531307 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:53.531209 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/409eabf1-9b70-47bc-a908-6f2d10477da3-webhook-cert\") pod \"maas-controller-6bc6856dbb-ftct4\" (UID: \"409eabf1-9b70-47bc-a908-6f2d10477da3\") " pod="opendatahub/maas-controller-6bc6856dbb-ftct4" Jul 10 20:46:53.531307 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:53.531248 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75dh6\" (UniqueName: \"kubernetes.io/projected/409eabf1-9b70-47bc-a908-6f2d10477da3-kube-api-access-75dh6\") pod \"maas-controller-6bc6856dbb-ftct4\" (UID: \"409eabf1-9b70-47bc-a908-6f2d10477da3\") " pod="opendatahub/maas-controller-6bc6856dbb-ftct4" Jul 10 20:46:53.632465 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:53.632427 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/409eabf1-9b70-47bc-a908-6f2d10477da3-webhook-cert\") pod \"maas-controller-6bc6856dbb-ftct4\" (UID: \"409eabf1-9b70-47bc-a908-6f2d10477da3\") " pod="opendatahub/maas-controller-6bc6856dbb-ftct4" Jul 10 20:46:53.632465 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:53.632464 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75dh6\" (UniqueName: \"kubernetes.io/projected/409eabf1-9b70-47bc-a908-6f2d10477da3-kube-api-access-75dh6\") pod \"maas-controller-6bc6856dbb-ftct4\" (UID: \"409eabf1-9b70-47bc-a908-6f2d10477da3\") " pod="opendatahub/maas-controller-6bc6856dbb-ftct4" Jul 10 20:46:53.635122 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:53.635085 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/409eabf1-9b70-47bc-a908-6f2d10477da3-webhook-cert\") pod \"maas-controller-6bc6856dbb-ftct4\" (UID: \"409eabf1-9b70-47bc-a908-6f2d10477da3\") " pod="opendatahub/maas-controller-6bc6856dbb-ftct4" Jul 10 20:46:53.643018 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:53.642989 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75dh6\" (UniqueName: \"kubernetes.io/projected/409eabf1-9b70-47bc-a908-6f2d10477da3-kube-api-access-75dh6\") pod \"maas-controller-6bc6856dbb-ftct4\" (UID: \"409eabf1-9b70-47bc-a908-6f2d10477da3\") " pod="opendatahub/maas-controller-6bc6856dbb-ftct4" Jul 10 20:46:53.759916 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:53.759871 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bc6856dbb-ftct4" Jul 10 20:46:53.896769 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:53.896733 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bc6856dbb-ftct4"] Jul 10 20:46:53.900357 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:46:53.900331 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409eabf1_9b70_47bc_a908_6f2d10477da3.slice/crio-e83c3e62137f259b884c0b85545489b43df9b02b5a49b2af11e99d5897d16166 WatchSource:0}: Error finding container e83c3e62137f259b884c0b85545489b43df9b02b5a49b2af11e99d5897d16166: Status 404 returned error can't find the container with id e83c3e62137f259b884c0b85545489b43df9b02b5a49b2af11e99d5897d16166 Jul 10 20:46:53.936093 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:53.936056 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f72372b4-db0c-4a0a-848a-14860e3bd9ff-webhook-cert\") pod \"maas-controller-686b46dd7f-4d7pv\" (UID: \"f72372b4-db0c-4a0a-848a-14860e3bd9ff\") " pod="opendatahub/maas-controller-686b46dd7f-4d7pv" Jul 10 20:46:53.938530 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:53.938501 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f72372b4-db0c-4a0a-848a-14860e3bd9ff-webhook-cert\") pod \"maas-controller-686b46dd7f-4d7pv\" (UID: \"f72372b4-db0c-4a0a-848a-14860e3bd9ff\") " pod="opendatahub/maas-controller-686b46dd7f-4d7pv" Jul 10 20:46:54.168425 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:54.168357 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-686b46dd7f-4d7pv" Jul 10 20:46:54.300288 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:54.300107 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-686b46dd7f-4d7pv"] Jul 10 20:46:54.302805 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:46:54.302777 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf72372b4_db0c_4a0a_848a_14860e3bd9ff.slice/crio-81c684cb6fde8c35cf675748de8e04909795db477c2d7823ecd44ef0c61ce03a WatchSource:0}: Error finding container 81c684cb6fde8c35cf675748de8e04909795db477c2d7823ecd44ef0c61ce03a: Status 404 returned error can't find the container with id 81c684cb6fde8c35cf675748de8e04909795db477c2d7823ecd44ef0c61ce03a Jul 10 20:46:54.437484 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:54.437383 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-686b46dd7f-4d7pv" event={"ID":"f72372b4-db0c-4a0a-848a-14860e3bd9ff","Type":"ContainerStarted","Data":"81c684cb6fde8c35cf675748de8e04909795db477c2d7823ecd44ef0c61ce03a"} Jul 10 20:46:54.438742 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:54.438707 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bc6856dbb-ftct4" event={"ID":"409eabf1-9b70-47bc-a908-6f2d10477da3","Type":"ContainerStarted","Data":"e83c3e62137f259b884c0b85545489b43df9b02b5a49b2af11e99d5897d16166"} Jul 10 20:46:57.454337 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:57.454300 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-686b46dd7f-4d7pv" event={"ID":"f72372b4-db0c-4a0a-848a-14860e3bd9ff","Type":"ContainerStarted","Data":"15c35ee6acfec5dd6290808857e8b9893105787dc196fabe7b73c51fe7e2f6ac"} Jul 10 20:46:57.454755 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:57.454479 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-686b46dd7f-4d7pv" Jul 10 20:46:57.455669 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:57.455648 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bc6856dbb-ftct4" event={"ID":"409eabf1-9b70-47bc-a908-6f2d10477da3","Type":"ContainerStarted","Data":"c2b8a06c8585ae23cd6ff67ccc7a23dc77f60baa222124d1a176efd67349882b"} Jul 10 20:46:57.455786 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:57.455773 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6bc6856dbb-ftct4" Jul 10 20:46:57.479754 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:57.479701 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-686b46dd7f-4d7pv" podStartSLOduration=2.30936528 podStartE2EDuration="4.479684512s" totalImagesPullingTime="2.170319232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:46:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:46:57.477967692 +0000 UTC m=+730.256549096" watchObservedRunningTime="2026-07-10 20:46:57.479684512 +0000 UTC m=+730.258265942" Jul 10 20:46:57.506106 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:46:57.506043 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6bc6856dbb-ftct4" podStartSLOduration=1.944460573 podStartE2EDuration="4.506020983s" totalImagesPullingTime="2.56156041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:46:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:46:57.503992075 +0000 UTC m=+730.282573470" watchObservedRunningTime="2026-07-10 20:46:57.506020983 +0000 UTC m=+730.284602379" Jul 10 20:47:08.465805 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:08.465773 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6bc6856dbb-ftct4" Jul 10 20:47:08.466230 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:08.465866 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-686b46dd7f-4d7pv" Jul 10 20:47:08.534749 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:08.534706 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-686b46dd7f-4d7pv"] Jul 10 20:47:08.534955 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:08.534911 2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-686b46dd7f-4d7pv" podUID="f72372b4-db0c-4a0a-848a-14860e3bd9ff" containerName="manager" containerID="cri-o://15c35ee6acfec5dd6290808857e8b9893105787dc196fabe7b73c51fe7e2f6ac" gracePeriod=30 Jul 10 20:47:08.776457 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:08.776431 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-686b46dd7f-4d7pv" Jul 10 20:47:08.867619 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:08.867579 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f72372b4-db0c-4a0a-848a-14860e3bd9ff-webhook-cert\") pod \"f72372b4-db0c-4a0a-848a-14860e3bd9ff\" (UID: \"f72372b4-db0c-4a0a-848a-14860e3bd9ff\") " Jul 10 20:47:08.867801 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:08.867634 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdffn\" (UniqueName: \"kubernetes.io/projected/f72372b4-db0c-4a0a-848a-14860e3bd9ff-kube-api-access-gdffn\") pod \"f72372b4-db0c-4a0a-848a-14860e3bd9ff\" (UID: \"f72372b4-db0c-4a0a-848a-14860e3bd9ff\") " Jul 10 20:47:08.869958 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:08.869932 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72372b4-db0c-4a0a-848a-14860e3bd9ff-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f72372b4-db0c-4a0a-848a-14860e3bd9ff" (UID: "f72372b4-db0c-4a0a-848a-14860e3bd9ff"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:47:08.870090 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:08.869963 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72372b4-db0c-4a0a-848a-14860e3bd9ff-kube-api-access-gdffn" (OuterVolumeSpecName: "kube-api-access-gdffn") pod "f72372b4-db0c-4a0a-848a-14860e3bd9ff" (UID: "f72372b4-db0c-4a0a-848a-14860e3bd9ff"). InnerVolumeSpecName "kube-api-access-gdffn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:47:08.968534 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:08.968495 2530 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f72372b4-db0c-4a0a-848a-14860e3bd9ff-webhook-cert\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 20:47:08.968534 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:08.968528 2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdffn\" (UniqueName: \"kubernetes.io/projected/f72372b4-db0c-4a0a-848a-14860e3bd9ff-kube-api-access-gdffn\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 20:47:09.502809 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:09.502773 2530 generic.go:356] "Generic (PLEG): container finished" podID="f72372b4-db0c-4a0a-848a-14860e3bd9ff" containerID="15c35ee6acfec5dd6290808857e8b9893105787dc196fabe7b73c51fe7e2f6ac" exitCode=0 Jul 10 20:47:09.503291 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:09.502819 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-686b46dd7f-4d7pv" Jul 10 20:47:09.503291 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:09.502823 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-686b46dd7f-4d7pv" event={"ID":"f72372b4-db0c-4a0a-848a-14860e3bd9ff","Type":"ContainerDied","Data":"15c35ee6acfec5dd6290808857e8b9893105787dc196fabe7b73c51fe7e2f6ac"} Jul 10 20:47:09.503291 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:09.502901 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-686b46dd7f-4d7pv" event={"ID":"f72372b4-db0c-4a0a-848a-14860e3bd9ff","Type":"ContainerDied","Data":"81c684cb6fde8c35cf675748de8e04909795db477c2d7823ecd44ef0c61ce03a"} Jul 10 20:47:09.503291 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:09.502935 2530 scope.go:117] "RemoveContainer" containerID="15c35ee6acfec5dd6290808857e8b9893105787dc196fabe7b73c51fe7e2f6ac" Jul 10 20:47:09.511259 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:09.511239 2530 scope.go:117] "RemoveContainer" containerID="15c35ee6acfec5dd6290808857e8b9893105787dc196fabe7b73c51fe7e2f6ac" Jul 10 20:47:09.511510 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:47:09.511493 2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c35ee6acfec5dd6290808857e8b9893105787dc196fabe7b73c51fe7e2f6ac\": container with ID starting with 15c35ee6acfec5dd6290808857e8b9893105787dc196fabe7b73c51fe7e2f6ac not found: ID does not exist" containerID="15c35ee6acfec5dd6290808857e8b9893105787dc196fabe7b73c51fe7e2f6ac" Jul 10 20:47:09.511553 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:09.511520 2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c35ee6acfec5dd6290808857e8b9893105787dc196fabe7b73c51fe7e2f6ac"} err="failed to get container status \"15c35ee6acfec5dd6290808857e8b9893105787dc196fabe7b73c51fe7e2f6ac\": rpc error: code = NotFound desc = could not find container \"15c35ee6acfec5dd6290808857e8b9893105787dc196fabe7b73c51fe7e2f6ac\": container with ID starting with 15c35ee6acfec5dd6290808857e8b9893105787dc196fabe7b73c51fe7e2f6ac not found: ID does not exist" Jul 10 20:47:09.544752 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:09.544709 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-686b46dd7f-4d7pv"] Jul 10 20:47:09.546444 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:09.546415 2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-686b46dd7f-4d7pv"] Jul 10 20:47:09.771786 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:09.771700 2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f72372b4-db0c-4a0a-848a-14860e3bd9ff" path="/var/lib/kubelet/pods/f72372b4-db0c-4a0a-848a-14860e3bd9ff/volumes" Jul 10 20:47:30.901976 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:30.901939 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-d59fd97f4-5k7v2"] Jul 10 20:47:30.902422 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:30.902268 2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f72372b4-db0c-4a0a-848a-14860e3bd9ff" containerName="manager" Jul 10 20:47:30.902422 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:30.902280 2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="f72372b4-db0c-4a0a-848a-14860e3bd9ff" containerName="manager" Jul 10 20:47:30.902422 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:30.902339 2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="f72372b4-db0c-4a0a-848a-14860e3bd9ff" containerName="manager" Jul 10 20:47:30.906431 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:30.906408 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d59fd97f4-5k7v2" Jul 10 20:47:30.913386 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:30.913361 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 10 20:47:30.933096 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:30.933064 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d59fd97f4-5k7v2"] Jul 10 20:47:31.051716 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.051674 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d8f1cc-7ae8-49ad-861f-ddeb0f64f4b6-openshift-service-ca\") pod \"maas-api-d59fd97f4-5k7v2\" (UID: \"63d8f1cc-7ae8-49ad-861f-ddeb0f64f4b6\") " pod="odh-ai-gateway-infra/maas-api-d59fd97f4-5k7v2" Jul 10 20:47:31.051920 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.051742 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8fq9\" (UniqueName: \"kubernetes.io/projected/63d8f1cc-7ae8-49ad-861f-ddeb0f64f4b6-kube-api-access-t8fq9\") pod \"maas-api-d59fd97f4-5k7v2\" (UID: \"63d8f1cc-7ae8-49ad-861f-ddeb0f64f4b6\") " pod="odh-ai-gateway-infra/maas-api-d59fd97f4-5k7v2" Jul 10 20:47:31.051920 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.051814 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d8f1cc-7ae8-49ad-861f-ddeb0f64f4b6-maas-api-tls\") pod \"maas-api-d59fd97f4-5k7v2\" (UID: \"63d8f1cc-7ae8-49ad-861f-ddeb0f64f4b6\") " pod="odh-ai-gateway-infra/maas-api-d59fd97f4-5k7v2" Jul 10 20:47:31.153244 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.153136 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d8f1cc-7ae8-49ad-861f-ddeb0f64f4b6-maas-api-tls\") pod \"maas-api-d59fd97f4-5k7v2\" (UID: \"63d8f1cc-7ae8-49ad-861f-ddeb0f64f4b6\") " pod="odh-ai-gateway-infra/maas-api-d59fd97f4-5k7v2" Jul 10 20:47:31.153244 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.153216 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d8f1cc-7ae8-49ad-861f-ddeb0f64f4b6-openshift-service-ca\") pod \"maas-api-d59fd97f4-5k7v2\" (UID: \"63d8f1cc-7ae8-49ad-861f-ddeb0f64f4b6\") " pod="odh-ai-gateway-infra/maas-api-d59fd97f4-5k7v2" Jul 10 20:47:31.153475 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.153263 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8fq9\" (UniqueName: \"kubernetes.io/projected/63d8f1cc-7ae8-49ad-861f-ddeb0f64f4b6-kube-api-access-t8fq9\") pod \"maas-api-d59fd97f4-5k7v2\" (UID: \"63d8f1cc-7ae8-49ad-861f-ddeb0f64f4b6\") " pod="odh-ai-gateway-infra/maas-api-d59fd97f4-5k7v2" Jul 10 20:47:31.153802 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.153777 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d8f1cc-7ae8-49ad-861f-ddeb0f64f4b6-openshift-service-ca\") pod \"maas-api-d59fd97f4-5k7v2\" (UID: \"63d8f1cc-7ae8-49ad-861f-ddeb0f64f4b6\") " pod="odh-ai-gateway-infra/maas-api-d59fd97f4-5k7v2" Jul 10 20:47:31.155642 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.155619 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d8f1cc-7ae8-49ad-861f-ddeb0f64f4b6-maas-api-tls\") pod \"maas-api-d59fd97f4-5k7v2\" (UID: \"63d8f1cc-7ae8-49ad-861f-ddeb0f64f4b6\") " pod="odh-ai-gateway-infra/maas-api-d59fd97f4-5k7v2" Jul 10 20:47:31.167054 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.167019 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8fq9\" (UniqueName: \"kubernetes.io/projected/63d8f1cc-7ae8-49ad-861f-ddeb0f64f4b6-kube-api-access-t8fq9\") pod \"maas-api-d59fd97f4-5k7v2\" (UID: \"63d8f1cc-7ae8-49ad-861f-ddeb0f64f4b6\") " pod="odh-ai-gateway-infra/maas-api-d59fd97f4-5k7v2" Jul 10 20:47:31.216748 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.216705 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d59fd97f4-5k7v2" Jul 10 20:47:31.357179 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.357147 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d59fd97f4-5k7v2"] Jul 10 20:47:31.360788 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:47:31.360759 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d8f1cc_7ae8_49ad_861f_ddeb0f64f4b6.slice/crio-1345bf4c14b500cd407b6faa8ce1c41d536c936c81545bb2a23596bc80fbeab5 WatchSource:0}: Error finding container 1345bf4c14b500cd407b6faa8ce1c41d536c936c81545bb2a23596bc80fbeab5: Status 404 returned error can't find the container with id 1345bf4c14b500cd407b6faa8ce1c41d536c936c81545bb2a23596bc80fbeab5 Jul 10 20:47:31.468007 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.467923 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-dxh94"] Jul 10 20:47:31.470931 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.470907 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-dxh94" Jul 10 20:47:31.473390 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.473367 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 10 20:47:31.485945 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.485919 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-dxh94"] Jul 10 20:47:31.508930 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.508898 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-ns2gt"] Jul 10 20:47:31.511223 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.511204 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-ns2gt" Jul 10 20:47:31.515251 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.515225 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-q7tbk\"" Jul 10 20:47:31.527235 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.527196 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-ns2gt"] Jul 10 20:47:31.588919 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.588889 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d59fd97f4-5k7v2" event={"ID":"63d8f1cc-7ae8-49ad-861f-ddeb0f64f4b6","Type":"ContainerStarted","Data":"1345bf4c14b500cd407b6faa8ce1c41d536c936c81545bb2a23596bc80fbeab5"} Jul 10 20:47:31.658246 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.658210 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bccfbea-5309-4201-8241-2a4236b00cf8-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-ns2gt\" (UID: \"5bccfbea-5309-4201-8241-2a4236b00cf8\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-ns2gt" Jul 10 20:47:31.658444 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.658265 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ddedd9-ceb8-4c42-9e9c-d760fed67935-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-dxh94\" (UID: \"a7ddedd9-ceb8-4c42-9e9c-d760fed67935\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-dxh94" Jul 10 20:47:31.658444 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.658343 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xmkk\" (UniqueName: \"kubernetes.io/projected/a7ddedd9-ceb8-4c42-9e9c-d760fed67935-kube-api-access-4xmkk\") pod \"payload-pre-processing-74584bb9f8-dxh94\" (UID: \"a7ddedd9-ceb8-4c42-9e9c-d760fed67935\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-dxh94" Jul 10 20:47:31.658444 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.658396 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr5rj\" (UniqueName: \"kubernetes.io/projected/5bccfbea-5309-4201-8241-2a4236b00cf8-kube-api-access-sr5rj\") pod \"payload-processing-f9b8d9b9b-ns2gt\" (UID: \"5bccfbea-5309-4201-8241-2a4236b00cf8\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-ns2gt" Jul 10 20:47:31.759662 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.759561 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xmkk\" (UniqueName: \"kubernetes.io/projected/a7ddedd9-ceb8-4c42-9e9c-d760fed67935-kube-api-access-4xmkk\") pod \"payload-pre-processing-74584bb9f8-dxh94\" (UID: \"a7ddedd9-ceb8-4c42-9e9c-d760fed67935\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-dxh94" Jul 10 20:47:31.759662 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.759613 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr5rj\" (UniqueName: \"kubernetes.io/projected/5bccfbea-5309-4201-8241-2a4236b00cf8-kube-api-access-sr5rj\") pod \"payload-processing-f9b8d9b9b-ns2gt\" (UID: \"5bccfbea-5309-4201-8241-2a4236b00cf8\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-ns2gt" Jul 10 20:47:31.759662 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.759649 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bccfbea-5309-4201-8241-2a4236b00cf8-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-ns2gt\" (UID: \"5bccfbea-5309-4201-8241-2a4236b00cf8\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-ns2gt" Jul 10 20:47:31.759982 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.759703 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ddedd9-ceb8-4c42-9e9c-d760fed67935-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-dxh94\" (UID: \"a7ddedd9-ceb8-4c42-9e9c-d760fed67935\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-dxh94" Jul 10 20:47:31.760449 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.760430 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ddedd9-ceb8-4c42-9e9c-d760fed67935-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-dxh94\" (UID: \"a7ddedd9-ceb8-4c42-9e9c-d760fed67935\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-dxh94" Jul 10 20:47:31.760487 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.760448 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bccfbea-5309-4201-8241-2a4236b00cf8-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-ns2gt\" (UID: \"5bccfbea-5309-4201-8241-2a4236b00cf8\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-ns2gt" Jul 10 20:47:31.769063 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.769026 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xmkk\" (UniqueName: \"kubernetes.io/projected/a7ddedd9-ceb8-4c42-9e9c-d760fed67935-kube-api-access-4xmkk\") pod \"payload-pre-processing-74584bb9f8-dxh94\" (UID: \"a7ddedd9-ceb8-4c42-9e9c-d760fed67935\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-dxh94" Jul 10 20:47:31.771986 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.771962 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr5rj\" (UniqueName: \"kubernetes.io/projected/5bccfbea-5309-4201-8241-2a4236b00cf8-kube-api-access-sr5rj\") pod \"payload-processing-f9b8d9b9b-ns2gt\" (UID: \"5bccfbea-5309-4201-8241-2a4236b00cf8\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-ns2gt" Jul 10 20:47:31.781802 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.781775 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-dxh94" Jul 10 20:47:31.821720 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.821683 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-ns2gt" Jul 10 20:47:31.925504 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.925477 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-dxh94"] Jul 10 20:47:31.927046 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:47:31.927012 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ddedd9_ceb8_4c42_9e9c_d760fed67935.slice/crio-4b6569485bc84cc2578163b235fe6d7ca89ee732356cfae8dead42e5f0a33e9d WatchSource:0}: Error finding container 4b6569485bc84cc2578163b235fe6d7ca89ee732356cfae8dead42e5f0a33e9d: Status 404 returned error can't find the container with id 4b6569485bc84cc2578163b235fe6d7ca89ee732356cfae8dead42e5f0a33e9d Jul 10 20:47:31.974992 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:31.974969 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-ns2gt"] Jul 10 20:47:31.977160 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:47:31.977126 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bccfbea_5309_4201_8241_2a4236b00cf8.slice/crio-1c159794a480861a427ab3655a5d2c0e44feef89706c733024af526b99b14286 WatchSource:0}: Error finding container 1c159794a480861a427ab3655a5d2c0e44feef89706c733024af526b99b14286: Status 404 returned error can't find the container with id 1c159794a480861a427ab3655a5d2c0e44feef89706c733024af526b99b14286 Jul 10 20:47:32.594998 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:32.594960 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-dxh94" event={"ID":"a7ddedd9-ceb8-4c42-9e9c-d760fed67935","Type":"ContainerStarted","Data":"4b6569485bc84cc2578163b235fe6d7ca89ee732356cfae8dead42e5f0a33e9d"} Jul 10 20:47:32.596338 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:32.596304 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-ns2gt" event={"ID":"5bccfbea-5309-4201-8241-2a4236b00cf8","Type":"ContainerStarted","Data":"1c159794a480861a427ab3655a5d2c0e44feef89706c733024af526b99b14286"} Jul 10 20:47:34.606179 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:34.606138 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-ns2gt" event={"ID":"5bccfbea-5309-4201-8241-2a4236b00cf8","Type":"ContainerStarted","Data":"4c8d24f3387a1013235688cf56c58eaaf5eaeb000dbbabfb792ad6c6fd887c9d"} Jul 10 20:47:34.606646 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:34.606204 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-ns2gt" Jul 10 20:47:34.607691 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:34.607667 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d59fd97f4-5k7v2" event={"ID":"63d8f1cc-7ae8-49ad-861f-ddeb0f64f4b6","Type":"ContainerStarted","Data":"56200a0e77494e8a4d9057bccfc06f24c9a0bb4a03fcef33602782b33dcd0f8f"} Jul 10 20:47:34.607812 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:34.607712 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-d59fd97f4-5k7v2" Jul 10 20:47:34.609105 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:34.609087 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-dxh94" event={"ID":"a7ddedd9-ceb8-4c42-9e9c-d760fed67935","Type":"ContainerStarted","Data":"c88c9671cd529bdadc66995a8d55ea87e7df917dbce2e3ae2791440d5c7bf915"} Jul 10 20:47:34.609221 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:34.609208 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-dxh94" Jul 10 20:47:34.627346 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:34.627190 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-f9b8d9b9b-ns2gt" podStartSLOduration=2.079239907 podStartE2EDuration="3.627169599s" totalImagesPullingTime="1.547929692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:47:34.626399609 +0000 UTC m=+767.404981002" watchObservedRunningTime="2026-07-10 20:47:34.627169599 +0000 UTC m=+767.405750993" Jul 10 20:47:34.649280 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:34.649215 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-dxh94" podStartSLOduration=2.052126066 podStartE2EDuration="3.649194337s" totalImagesPullingTime="1.597068271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:47:34.648646985 +0000 UTC m=+767.427228379" watchObservedRunningTime="2026-07-10 20:47:34.649194337 +0000 UTC m=+767.427775713" Jul 10 20:47:34.671976 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:34.671910 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-d59fd97f4-5k7v2" podStartSLOduration=2.507767069 podStartE2EDuration="4.671888356s" totalImagesPullingTime="2.164121287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:47:34.668703336 +0000 UTC m=+767.447284753" watchObservedRunningTime="2026-07-10 20:47:34.671888356 +0000 UTC m=+767.450469751" Jul 10 20:47:40.619217 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:40.619187 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-d59fd97f4-5k7v2" Jul 10 20:47:45.616200 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:45.616167 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-ns2gt" Jul 10 20:47:45.616683 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:47:45.616238 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-dxh94" Jul 10 20:48:02.527424 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.527386 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd"] Jul 10 20:48:02.531981 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.531958 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" Jul 10 20:48:02.534579 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.534555 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 20:48:02.535290 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.535270 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 10 20:48:02.535290 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.535282 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 20:48:02.535436 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.535309 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-szg4s\"" Jul 10 20:48:02.543031 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.543009 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd"] Jul 10 20:48:02.607916 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.607877 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23d1b156-b04a-417e-be23-0d58b14cf223-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cfbjd\" (UID: \"23d1b156-b04a-417e-be23-0d58b14cf223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" Jul 10 20:48:02.608101 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.607985 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23d1b156-b04a-417e-be23-0d58b14cf223-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cfbjd\" (UID: \"23d1b156-b04a-417e-be23-0d58b14cf223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" Jul 10 20:48:02.608101 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.608035 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6vvw\" (UniqueName: \"kubernetes.io/projected/23d1b156-b04a-417e-be23-0d58b14cf223-kube-api-access-v6vvw\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cfbjd\" (UID: \"23d1b156-b04a-417e-be23-0d58b14cf223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" Jul 10 20:48:02.608198 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.608101 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23d1b156-b04a-417e-be23-0d58b14cf223-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cfbjd\" (UID: \"23d1b156-b04a-417e-be23-0d58b14cf223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" Jul 10 20:48:02.608198 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.608156 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23d1b156-b04a-417e-be23-0d58b14cf223-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cfbjd\" (UID: \"23d1b156-b04a-417e-be23-0d58b14cf223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" Jul 10 20:48:02.608198 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.608174 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23d1b156-b04a-417e-be23-0d58b14cf223-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cfbjd\" (UID: \"23d1b156-b04a-417e-be23-0d58b14cf223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" Jul 10 20:48:02.709439 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.709402 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23d1b156-b04a-417e-be23-0d58b14cf223-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cfbjd\" (UID: \"23d1b156-b04a-417e-be23-0d58b14cf223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" Jul 10 20:48:02.709628 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.709463 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23d1b156-b04a-417e-be23-0d58b14cf223-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cfbjd\" (UID: \"23d1b156-b04a-417e-be23-0d58b14cf223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" Jul 10 20:48:02.709628 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.709497 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6vvw\" (UniqueName: \"kubernetes.io/projected/23d1b156-b04a-417e-be23-0d58b14cf223-kube-api-access-v6vvw\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cfbjd\" (UID: \"23d1b156-b04a-417e-be23-0d58b14cf223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" Jul 10 20:48:02.709628 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.709540 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23d1b156-b04a-417e-be23-0d58b14cf223-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cfbjd\" (UID: \"23d1b156-b04a-417e-be23-0d58b14cf223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" Jul 10 20:48:02.709628 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.709589 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23d1b156-b04a-417e-be23-0d58b14cf223-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cfbjd\" (UID: \"23d1b156-b04a-417e-be23-0d58b14cf223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" Jul 10 20:48:02.709628 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.709611 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23d1b156-b04a-417e-be23-0d58b14cf223-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cfbjd\" (UID: \"23d1b156-b04a-417e-be23-0d58b14cf223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" Jul 10 20:48:02.709997 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.709971 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23d1b156-b04a-417e-be23-0d58b14cf223-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cfbjd\" (UID: \"23d1b156-b04a-417e-be23-0d58b14cf223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" Jul 10 20:48:02.710064 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.710016 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23d1b156-b04a-417e-be23-0d58b14cf223-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cfbjd\" (UID: \"23d1b156-b04a-417e-be23-0d58b14cf223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" Jul 10 20:48:02.710207 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.710179 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23d1b156-b04a-417e-be23-0d58b14cf223-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cfbjd\" (UID: \"23d1b156-b04a-417e-be23-0d58b14cf223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" Jul 10 20:48:02.711696 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.711678 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23d1b156-b04a-417e-be23-0d58b14cf223-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cfbjd\" (UID: \"23d1b156-b04a-417e-be23-0d58b14cf223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" Jul 10 20:48:02.712004 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.711987 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23d1b156-b04a-417e-be23-0d58b14cf223-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cfbjd\" (UID: \"23d1b156-b04a-417e-be23-0d58b14cf223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" Jul 10 20:48:02.717704 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.717673 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6vvw\" (UniqueName: \"kubernetes.io/projected/23d1b156-b04a-417e-be23-0d58b14cf223-kube-api-access-v6vvw\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cfbjd\" (UID: \"23d1b156-b04a-417e-be23-0d58b14cf223\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" Jul 10 20:48:02.844578 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.844489 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" Jul 10 20:48:02.974415 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:02.974393 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd"] Jul 10 20:48:02.976694 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:48:02.976666 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d1b156_b04a_417e_be23_0d58b14cf223.slice/crio-b922df0277b8caf4b802fbf5954c7c56772b9daec62cdeeb950b7c7577386a8a WatchSource:0}: Error finding container b922df0277b8caf4b802fbf5954c7c56772b9daec62cdeeb950b7c7577386a8a: Status 404 returned error can't find the container with id b922df0277b8caf4b802fbf5954c7c56772b9daec62cdeeb950b7c7577386a8a Jul 10 20:48:03.731165 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:03.731127 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" event={"ID":"23d1b156-b04a-417e-be23-0d58b14cf223","Type":"ContainerStarted","Data":"b922df0277b8caf4b802fbf5954c7c56772b9daec62cdeeb950b7c7577386a8a"} Jul 10 20:48:04.736882 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:04.736822 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" event={"ID":"23d1b156-b04a-417e-be23-0d58b14cf223","Type":"ContainerStarted","Data":"84d2cb81aa74906afc86f18c1111f342e06aaac7860ce6bbd2a50394f4ef516b"} Jul 10 20:48:04.762105 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:04.762053 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" podStartSLOduration=1.74073243 podStartE2EDuration="2.762035332s" totalImagesPullingTime="1.021302902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:48:04.760881607 +0000 UTC m=+797.539463001" watchObservedRunningTime="2026-07-10 20:48:04.762035332 +0000 UTC m=+797.540616726" Jul 10 20:48:08.427569 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:08.427539 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x"] Jul 10 20:48:08.431760 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:08.431740 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" Jul 10 20:48:08.434872 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:08.434848 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 10 20:48:08.441677 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:08.441657 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x"] Jul 10 20:48:08.563378 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:08.563337 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23d257b-40b9-469f-868d-0715c3006b72-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv75x\" (UID: \"e23d257b-40b9-469f-868d-0715c3006b72\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" Jul 10 20:48:08.563553 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:08.563394 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23d257b-40b9-469f-868d-0715c3006b72-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv75x\" (UID: \"e23d257b-40b9-469f-868d-0715c3006b72\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" Jul 10 20:48:08.563553 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:08.563451 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7njp\" (UniqueName: \"kubernetes.io/projected/e23d257b-40b9-469f-868d-0715c3006b72-kube-api-access-h7njp\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv75x\" (UID: \"e23d257b-40b9-469f-868d-0715c3006b72\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" Jul 10 20:48:08.563553 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:08.563476 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e23d257b-40b9-469f-868d-0715c3006b72-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv75x\" (UID: \"e23d257b-40b9-469f-868d-0715c3006b72\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" Jul 10 20:48:08.563553 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:08.563532 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e23d257b-40b9-469f-868d-0715c3006b72-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv75x\" (UID: \"e23d257b-40b9-469f-868d-0715c3006b72\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" Jul 10 20:48:08.563686 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:08.563591 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e23d257b-40b9-469f-868d-0715c3006b72-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv75x\" (UID: \"e23d257b-40b9-469f-868d-0715c3006b72\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" Jul 10 20:48:08.665025 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:08.664987 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e23d257b-40b9-469f-868d-0715c3006b72-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv75x\" (UID: \"e23d257b-40b9-469f-868d-0715c3006b72\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" Jul 10 20:48:08.665025 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:08.665027 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23d257b-40b9-469f-868d-0715c3006b72-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv75x\" (UID: \"e23d257b-40b9-469f-868d-0715c3006b72\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" Jul 10 20:48:08.665275 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:08.665068 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23d257b-40b9-469f-868d-0715c3006b72-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv75x\" (UID: \"e23d257b-40b9-469f-868d-0715c3006b72\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" Jul 10 20:48:08.665275 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:08.665108 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7njp\" (UniqueName: \"kubernetes.io/projected/e23d257b-40b9-469f-868d-0715c3006b72-kube-api-access-h7njp\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv75x\" (UID: \"e23d257b-40b9-469f-868d-0715c3006b72\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" Jul 10 20:48:08.665275 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:08.665142 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e23d257b-40b9-469f-868d-0715c3006b72-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv75x\" (UID: \"e23d257b-40b9-469f-868d-0715c3006b72\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" Jul 10 20:48:08.665275 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:08.665186 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e23d257b-40b9-469f-868d-0715c3006b72-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv75x\" (UID: \"e23d257b-40b9-469f-868d-0715c3006b72\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" Jul 10 20:48:08.665500 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:08.665435 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e23d257b-40b9-469f-868d-0715c3006b72-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv75x\" (UID: \"e23d257b-40b9-469f-868d-0715c3006b72\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" Jul 10 20:48:08.665558 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:08.665542 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e23d257b-40b9-469f-868d-0715c3006b72-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv75x\" (UID: \"e23d257b-40b9-469f-868d-0715c3006b72\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" Jul 10 20:48:08.665558 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:08.665543 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23d257b-40b9-469f-868d-0715c3006b72-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv75x\" (UID: \"e23d257b-40b9-469f-868d-0715c3006b72\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" Jul 10 20:48:08.667479 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:08.667455 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e23d257b-40b9-469f-868d-0715c3006b72-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv75x\" (UID: \"e23d257b-40b9-469f-868d-0715c3006b72\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" Jul 10 20:48:08.667698 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:08.667682 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23d257b-40b9-469f-868d-0715c3006b72-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv75x\" (UID: \"e23d257b-40b9-469f-868d-0715c3006b72\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" Jul 10 20:48:08.674191 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:08.674153 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7njp\" (UniqueName: \"kubernetes.io/projected/e23d257b-40b9-469f-868d-0715c3006b72-kube-api-access-h7njp\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sv75x\" (UID: \"e23d257b-40b9-469f-868d-0715c3006b72\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" Jul 10 20:48:08.742471 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:08.742366 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" Jul 10 20:48:08.875176 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:08.875131 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x"] Jul 10 20:48:08.877826 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:48:08.877800 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23d257b_40b9_469f_868d_0715c3006b72.slice/crio-c46ad5d667f9480e0b355c665f103787e705fd03ceabadbdb4d3dd805527ae39 WatchSource:0}: Error finding container c46ad5d667f9480e0b355c665f103787e705fd03ceabadbdb4d3dd805527ae39: Status 404 returned error can't find the container with id c46ad5d667f9480e0b355c665f103787e705fd03ceabadbdb4d3dd805527ae39 Jul 10 20:48:09.758529 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:09.758493 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" event={"ID":"e23d257b-40b9-469f-868d-0715c3006b72","Type":"ContainerStarted","Data":"95923120cd0c29d840a5507ea764bb8e0d7858100a8e1474e96f4160bb8c7b9f"} Jul 10 20:48:09.758933 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:09.758549 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" event={"ID":"e23d257b-40b9-469f-868d-0715c3006b72","Type":"ContainerStarted","Data":"c46ad5d667f9480e0b355c665f103787e705fd03ceabadbdb4d3dd805527ae39"} Jul 10 20:48:09.790942 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:09.790888 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" podStartSLOduration=1.604437187 podStartE2EDuration="1.790872887s" totalImagesPullingTime="186.4357ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:48:09.790035613 +0000 UTC m=+802.568617018" watchObservedRunningTime="2026-07-10 20:48:09.790872887 +0000 UTC m=+802.569454280" Jul 10 20:48:12.845117 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:12.845073 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" Jul 10 20:48:12.845623 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:12.845149 2530 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" Jul 10 20:48:12.857693 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:12.857663 2530 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" Jul 10 20:48:13.787269 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:13.787241 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cfbjd" Jul 10 20:48:18.743451 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:18.743409 2530 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" Jul 10 20:48:18.744008 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:18.743494 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" Jul 10 20:48:18.755780 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:18.755754 2530 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" Jul 10 20:48:18.809539 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:48:18.809506 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sv75x" Jul 10 20:49:47.704042 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:49:47.703965 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n7kg6_a0fc144e-a4ca-4f83-9126-9366a781f0ad/ovn-acl-logging/0.log" Jul 10 20:49:47.706744 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:49:47.706711 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n7kg6_a0fc144e-a4ca-4f83-9126-9366a781f0ad/ovn-acl-logging/0.log" Jul 10 20:51:14.262627 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:14.262534 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6bc6856dbb-ftct4"] Jul 10 20:51:14.263191 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:14.262811 2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6bc6856dbb-ftct4" podUID="409eabf1-9b70-47bc-a908-6f2d10477da3" containerName="manager" containerID="cri-o://c2b8a06c8585ae23cd6ff67ccc7a23dc77f60baa222124d1a176efd67349882b" gracePeriod=30 Jul 10 20:51:14.509220 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:14.509193 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bc6856dbb-ftct4" Jul 10 20:51:14.552642 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:14.552550 2530 generic.go:356] "Generic (PLEG): container finished" podID="409eabf1-9b70-47bc-a908-6f2d10477da3" containerID="c2b8a06c8585ae23cd6ff67ccc7a23dc77f60baa222124d1a176efd67349882b" exitCode=0 Jul 10 20:51:14.552642 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:14.552599 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bc6856dbb-ftct4" Jul 10 20:51:14.552642 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:14.552632 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bc6856dbb-ftct4" event={"ID":"409eabf1-9b70-47bc-a908-6f2d10477da3","Type":"ContainerDied","Data":"c2b8a06c8585ae23cd6ff67ccc7a23dc77f60baa222124d1a176efd67349882b"} Jul 10 20:51:14.552894 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:14.552676 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bc6856dbb-ftct4" event={"ID":"409eabf1-9b70-47bc-a908-6f2d10477da3","Type":"ContainerDied","Data":"e83c3e62137f259b884c0b85545489b43df9b02b5a49b2af11e99d5897d16166"} Jul 10 20:51:14.552894 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:14.552702 2530 scope.go:117] "RemoveContainer" containerID="c2b8a06c8585ae23cd6ff67ccc7a23dc77f60baa222124d1a176efd67349882b" Jul 10 20:51:14.560560 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:14.560536 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/409eabf1-9b70-47bc-a908-6f2d10477da3-webhook-cert\") pod \"409eabf1-9b70-47bc-a908-6f2d10477da3\" (UID: \"409eabf1-9b70-47bc-a908-6f2d10477da3\") " Jul 10 20:51:14.560692 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:14.560627 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75dh6\" (UniqueName: \"kubernetes.io/projected/409eabf1-9b70-47bc-a908-6f2d10477da3-kube-api-access-75dh6\") pod \"409eabf1-9b70-47bc-a908-6f2d10477da3\" (UID: \"409eabf1-9b70-47bc-a908-6f2d10477da3\") " Jul 10 20:51:14.561128 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:14.561109 2530 scope.go:117] "RemoveContainer" containerID="c2b8a06c8585ae23cd6ff67ccc7a23dc77f60baa222124d1a176efd67349882b" Jul 10 20:51:14.561471 ip-10-0-139-64 kubenswrapper[2530]: E0710 20:51:14.561447 2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2b8a06c8585ae23cd6ff67ccc7a23dc77f60baa222124d1a176efd67349882b\": container with ID starting with c2b8a06c8585ae23cd6ff67ccc7a23dc77f60baa222124d1a176efd67349882b not found: ID does not exist" containerID="c2b8a06c8585ae23cd6ff67ccc7a23dc77f60baa222124d1a176efd67349882b" Jul 10 20:51:14.561565 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:14.561477 2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2b8a06c8585ae23cd6ff67ccc7a23dc77f60baa222124d1a176efd67349882b"} err="failed to get container status \"c2b8a06c8585ae23cd6ff67ccc7a23dc77f60baa222124d1a176efd67349882b\": rpc error: code = NotFound desc = could not find container \"c2b8a06c8585ae23cd6ff67ccc7a23dc77f60baa222124d1a176efd67349882b\": container with ID starting with c2b8a06c8585ae23cd6ff67ccc7a23dc77f60baa222124d1a176efd67349882b not found: ID does not exist" Jul 10 20:51:14.562746 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:14.562720 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/409eabf1-9b70-47bc-a908-6f2d10477da3-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "409eabf1-9b70-47bc-a908-6f2d10477da3" (UID: "409eabf1-9b70-47bc-a908-6f2d10477da3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:51:14.562848 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:14.562749 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409eabf1-9b70-47bc-a908-6f2d10477da3-kube-api-access-75dh6" (OuterVolumeSpecName: "kube-api-access-75dh6") pod "409eabf1-9b70-47bc-a908-6f2d10477da3" (UID: "409eabf1-9b70-47bc-a908-6f2d10477da3"). InnerVolumeSpecName "kube-api-access-75dh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:51:14.662012 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:14.661958 2530 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/409eabf1-9b70-47bc-a908-6f2d10477da3-webhook-cert\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 20:51:14.662012 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:14.662002 2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75dh6\" (UniqueName: \"kubernetes.io/projected/409eabf1-9b70-47bc-a908-6f2d10477da3-kube-api-access-75dh6\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 20:51:14.874496 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:14.874464 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6bc6856dbb-ftct4"] Jul 10 20:51:14.878352 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:14.878322 2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6bc6856dbb-ftct4"] Jul 10 20:51:15.676076 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:15.676045 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6bc6856dbb-pr9zj"] Jul 10 20:51:15.676464 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:15.676350 2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="409eabf1-9b70-47bc-a908-6f2d10477da3" containerName="manager" Jul 10 20:51:15.676464 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:15.676362 2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="409eabf1-9b70-47bc-a908-6f2d10477da3" containerName="manager" Jul 10 20:51:15.676464 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:15.676422 2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="409eabf1-9b70-47bc-a908-6f2d10477da3" containerName="manager" Jul 10 20:51:15.678929 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:15.678910 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bc6856dbb-pr9zj" Jul 10 20:51:15.683112 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:15.683091 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 20:51:15.683250 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:15.683127 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pd7v2\"" Jul 10 20:51:15.683481 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:15.683468 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 20:51:15.698497 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:15.698470 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bc6856dbb-pr9zj"] Jul 10 20:51:15.771479 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:15.771451 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbc0bae2-e7f8-446d-b6a5-f99bdc987030-webhook-cert\") pod \"maas-controller-6bc6856dbb-pr9zj\" (UID: \"fbc0bae2-e7f8-446d-b6a5-f99bdc987030\") " pod="opendatahub/maas-controller-6bc6856dbb-pr9zj" Jul 10 20:51:15.771646 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:15.771485 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrpcr\" (UniqueName: \"kubernetes.io/projected/fbc0bae2-e7f8-446d-b6a5-f99bdc987030-kube-api-access-hrpcr\") pod \"maas-controller-6bc6856dbb-pr9zj\" (UID: \"fbc0bae2-e7f8-446d-b6a5-f99bdc987030\") " pod="opendatahub/maas-controller-6bc6856dbb-pr9zj" Jul 10 20:51:15.772294 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:15.772269 2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="409eabf1-9b70-47bc-a908-6f2d10477da3" path="/var/lib/kubelet/pods/409eabf1-9b70-47bc-a908-6f2d10477da3/volumes" Jul 10 20:51:15.872491 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:15.872455 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbc0bae2-e7f8-446d-b6a5-f99bdc987030-webhook-cert\") pod \"maas-controller-6bc6856dbb-pr9zj\" (UID: \"fbc0bae2-e7f8-446d-b6a5-f99bdc987030\") " pod="opendatahub/maas-controller-6bc6856dbb-pr9zj" Jul 10 20:51:15.872635 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:15.872498 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrpcr\" (UniqueName: \"kubernetes.io/projected/fbc0bae2-e7f8-446d-b6a5-f99bdc987030-kube-api-access-hrpcr\") pod \"maas-controller-6bc6856dbb-pr9zj\" (UID: \"fbc0bae2-e7f8-446d-b6a5-f99bdc987030\") " pod="opendatahub/maas-controller-6bc6856dbb-pr9zj" Jul 10 20:51:15.874921 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:15.874891 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbc0bae2-e7f8-446d-b6a5-f99bdc987030-webhook-cert\") pod \"maas-controller-6bc6856dbb-pr9zj\" (UID: \"fbc0bae2-e7f8-446d-b6a5-f99bdc987030\") " pod="opendatahub/maas-controller-6bc6856dbb-pr9zj" Jul 10 20:51:15.881641 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:15.881611 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrpcr\" (UniqueName: \"kubernetes.io/projected/fbc0bae2-e7f8-446d-b6a5-f99bdc987030-kube-api-access-hrpcr\") pod \"maas-controller-6bc6856dbb-pr9zj\" (UID: \"fbc0bae2-e7f8-446d-b6a5-f99bdc987030\") " pod="opendatahub/maas-controller-6bc6856dbb-pr9zj" Jul 10 20:51:15.989067 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:15.988976 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bc6856dbb-pr9zj" Jul 10 20:51:16.119027 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:16.118963 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bc6856dbb-pr9zj"] Jul 10 20:51:16.121380 ip-10-0-139-64 kubenswrapper[2530]: W0710 20:51:16.121350 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc0bae2_e7f8_446d_b6a5_f99bdc987030.slice/crio-522d02bc43d991d05f2b384821f3795e21d5be9b36bb809fe2d9a9a15722cd6b WatchSource:0}: Error finding container 522d02bc43d991d05f2b384821f3795e21d5be9b36bb809fe2d9a9a15722cd6b: Status 404 returned error can't find the container with id 522d02bc43d991d05f2b384821f3795e21d5be9b36bb809fe2d9a9a15722cd6b Jul 10 20:51:16.122791 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:16.122771 2530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 20:51:16.563621 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:16.563589 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bc6856dbb-pr9zj" event={"ID":"fbc0bae2-e7f8-446d-b6a5-f99bdc987030","Type":"ContainerStarted","Data":"522d02bc43d991d05f2b384821f3795e21d5be9b36bb809fe2d9a9a15722cd6b"} Jul 10 20:51:17.569464 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:17.569431 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bc6856dbb-pr9zj" event={"ID":"fbc0bae2-e7f8-446d-b6a5-f99bdc987030","Type":"ContainerStarted","Data":"7f9ec0056d873b2d902268c9512b2397f4b2cca5d38dfd86c5cbb14651889771"} Jul 10 20:51:17.569902 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:17.569538 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6bc6856dbb-pr9zj" Jul 10 20:51:17.598790 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:17.598727 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6bc6856dbb-pr9zj" podStartSLOduration=2.15336485 podStartE2EDuration="2.598710792s" totalImagesPullingTime="445.345942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:51:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:51:17.597186827 +0000 UTC m=+990.375768208" watchObservedRunningTime="2026-07-10 20:51:17.598710792 +0000 UTC m=+990.377292221" Jul 10 20:51:28.581184 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:51:28.581142 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6bc6856dbb-pr9zj" Jul 10 20:54:47.728413 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:54:47.728381 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n7kg6_a0fc144e-a4ca-4f83-9126-9366a781f0ad/ovn-acl-logging/0.log" Jul 10 20:54:47.731494 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:54:47.731471 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n7kg6_a0fc144e-a4ca-4f83-9126-9366a781f0ad/ovn-acl-logging/0.log" Jul 10 20:59:47.755188 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:59:47.755157 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n7kg6_a0fc144e-a4ca-4f83-9126-9366a781f0ad/ovn-acl-logging/0.log" Jul 10 20:59:47.758694 ip-10-0-139-64 kubenswrapper[2530]: I0710 20:59:47.758673 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n7kg6_a0fc144e-a4ca-4f83-9126-9366a781f0ad/ovn-acl-logging/0.log" Jul 10 21:00:00.148128 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:00:00.148092 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728620-7jmd2"] Jul 10 21:00:00.151395 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:00:00.151377 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728620-7jmd2" Jul 10 21:00:00.153786 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:00:00.153766 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b692z\"" Jul 10 21:00:00.161001 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:00:00.160981 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728620-7jmd2" Jul 10 21:00:00.168275 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:00:00.168250 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728620-7jmd2"] Jul 10 21:00:00.291184 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:00:00.291156 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728620-7jmd2"] Jul 10 21:00:00.293599 ip-10-0-139-64 kubenswrapper[2530]: W0710 21:00:00.293567 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ef09b4_fca2_4401_b4c4_4c8a8cb0053a.slice/crio-1a8257ac9fefe66514608b65d53ec753e947817b1a0f0e93f26e38a6db01df54 WatchSource:0}: Error finding container 1a8257ac9fefe66514608b65d53ec753e947817b1a0f0e93f26e38a6db01df54: Status 404 returned error can't find the container with id 1a8257ac9fefe66514608b65d53ec753e947817b1a0f0e93f26e38a6db01df54 Jul 10 21:00:00.295634 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:00:00.295615 2530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 21:00:00.772632 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:00:00.772595 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728620-7jmd2" event={"ID":"35ef09b4-fca2-4401-b4c4-4c8a8cb0053a","Type":"ContainerStarted","Data":"1a8257ac9fefe66514608b65d53ec753e947817b1a0f0e93f26e38a6db01df54"} Jul 10 21:00:11.820216 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:00:11.820177 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728620-7jmd2" event={"ID":"35ef09b4-fca2-4401-b4c4-4c8a8cb0053a","Type":"ContainerStarted","Data":"63292428b49ef158ef9fd8d888daabec75d1a876855b680f97f631420b350eeb"} Jul 10 21:00:11.837308 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:00:11.837253 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728620-7jmd2" podStartSLOduration=0.888947823 podStartE2EDuration="11.837232019s" totalImagesPullingTime="10.948284196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:00:11.836594913 +0000 UTC m=+1524.615176307" watchObservedRunningTime="2026-07-10 21:00:11.837232019 +0000 UTC m=+1524.615813414" Jul 10 21:01:00.010228 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:01:00.010199 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728620-7jmd2"] Jul 10 21:01:00.010741 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:01:00.010419 2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728620-7jmd2" podUID="35ef09b4-fca2-4401-b4c4-4c8a8cb0053a" containerName="cleanup" containerID="cri-o://63292428b49ef158ef9fd8d888daabec75d1a876855b680f97f631420b350eeb" gracePeriod=30 Jul 10 21:01:30.156738 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:01:30.156703 2530 generic.go:356] "Generic (PLEG): container finished" podID="35ef09b4-fca2-4401-b4c4-4c8a8cb0053a" containerID="63292428b49ef158ef9fd8d888daabec75d1a876855b680f97f631420b350eeb" exitCode=137 Jul 10 21:01:30.157153 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:01:30.156774 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728620-7jmd2" event={"ID":"35ef09b4-fca2-4401-b4c4-4c8a8cb0053a","Type":"ContainerDied","Data":"63292428b49ef158ef9fd8d888daabec75d1a876855b680f97f631420b350eeb"} Jul 10 21:01:30.188477 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:01:30.188450 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728620-7jmd2" Jul 10 21:01:31.162280 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:01:31.162252 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728620-7jmd2" Jul 10 21:01:31.162765 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:01:31.162245 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728620-7jmd2" event={"ID":"35ef09b4-fca2-4401-b4c4-4c8a8cb0053a","Type":"ContainerDied","Data":"1a8257ac9fefe66514608b65d53ec753e947817b1a0f0e93f26e38a6db01df54"} Jul 10 21:01:31.162765 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:01:31.162381 2530 scope.go:117] "RemoveContainer" containerID="63292428b49ef158ef9fd8d888daabec75d1a876855b680f97f631420b350eeb" Jul 10 21:01:31.190163 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:01:31.190122 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728620-7jmd2"] Jul 10 21:01:31.193429 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:01:31.193403 2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728620-7jmd2"] Jul 10 21:01:31.771871 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:01:31.771822 2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35ef09b4-fca2-4401-b4c4-4c8a8cb0053a" path="/var/lib/kubelet/pods/35ef09b4-fca2-4401-b4c4-4c8a8cb0053a/volumes" Jul 10 21:03:28.797621 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:28.797584 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz"] Jul 10 21:03:28.798167 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:28.798149 2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35ef09b4-fca2-4401-b4c4-4c8a8cb0053a" containerName="cleanup" Jul 10 21:03:28.798236 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:28.798172 2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="35ef09b4-fca2-4401-b4c4-4c8a8cb0053a" containerName="cleanup" Jul 10 21:03:28.798284 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:28.798262 2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="35ef09b4-fca2-4401-b4c4-4c8a8cb0053a" containerName="cleanup" Jul 10 21:03:28.801414 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:28.801392 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz" Jul 10 21:03:28.804223 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:28.804190 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pm7gf\"" Jul 10 21:03:28.804223 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:28.804201 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6882a6b6-serving-cert\"" Jul 10 21:03:28.814384 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:28.814357 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz"] Jul 10 21:03:28.925089 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:28.925059 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58c0a40d-da09-45c4-83c8-49cba7d91b68-maas-api-tls\") pod \"maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz\" (UID: \"58c0a40d-da09-45c4-83c8-49cba7d91b68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz" Jul 10 21:03:28.925288 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:28.925113 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4gwz\" (UniqueName: \"kubernetes.io/projected/58c0a40d-da09-45c4-83c8-49cba7d91b68-kube-api-access-c4gwz\") pod \"maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz\" (UID: \"58c0a40d-da09-45c4-83c8-49cba7d91b68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz" Jul 10 21:03:28.925288 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:28.925155 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58c0a40d-da09-45c4-83c8-49cba7d91b68-openshift-service-ca\") pod \"maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz\" (UID: \"58c0a40d-da09-45c4-83c8-49cba7d91b68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz" Jul 10 21:03:29.026232 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:29.026190 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58c0a40d-da09-45c4-83c8-49cba7d91b68-openshift-service-ca\") pod \"maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz\" (UID: \"58c0a40d-da09-45c4-83c8-49cba7d91b68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz" Jul 10 21:03:29.026424 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:29.026251 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58c0a40d-da09-45c4-83c8-49cba7d91b68-maas-api-tls\") pod \"maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz\" (UID: \"58c0a40d-da09-45c4-83c8-49cba7d91b68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz" Jul 10 21:03:29.026424 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:29.026304 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4gwz\" (UniqueName: \"kubernetes.io/projected/58c0a40d-da09-45c4-83c8-49cba7d91b68-kube-api-access-c4gwz\") pod \"maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz\" (UID: \"58c0a40d-da09-45c4-83c8-49cba7d91b68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz" Jul 10 21:03:29.026984 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:29.026959 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58c0a40d-da09-45c4-83c8-49cba7d91b68-openshift-service-ca\") pod \"maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz\" (UID: \"58c0a40d-da09-45c4-83c8-49cba7d91b68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz" Jul 10 21:03:29.028679 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:29.028653 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58c0a40d-da09-45c4-83c8-49cba7d91b68-maas-api-tls\") pod \"maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz\" (UID: \"58c0a40d-da09-45c4-83c8-49cba7d91b68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz" Jul 10 21:03:29.035015 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:29.034992 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4gwz\" (UniqueName: \"kubernetes.io/projected/58c0a40d-da09-45c4-83c8-49cba7d91b68-kube-api-access-c4gwz\") pod \"maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz\" (UID: \"58c0a40d-da09-45c4-83c8-49cba7d91b68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz" Jul 10 21:03:29.113097 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:29.113009 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz" Jul 10 21:03:29.257106 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:29.257079 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz"] Jul 10 21:03:29.258811 ip-10-0-139-64 kubenswrapper[2530]: W0710 21:03:29.258781 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c0a40d_da09_45c4_83c8_49cba7d91b68.slice/crio-bac6bc3492cd3aca2337a44b45b2233da93d11b6dd3a1c3355f13d651d3da04f WatchSource:0}: Error finding container bac6bc3492cd3aca2337a44b45b2233da93d11b6dd3a1c3355f13d651d3da04f: Status 404 returned error can't find the container with id bac6bc3492cd3aca2337a44b45b2233da93d11b6dd3a1c3355f13d651d3da04f Jul 10 21:03:29.671219 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:29.671176 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz" event={"ID":"58c0a40d-da09-45c4-83c8-49cba7d91b68","Type":"ContainerStarted","Data":"bac6bc3492cd3aca2337a44b45b2233da93d11b6dd3a1c3355f13d651d3da04f"} Jul 10 21:03:30.676457 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:30.676424 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz" event={"ID":"58c0a40d-da09-45c4-83c8-49cba7d91b68","Type":"ContainerStarted","Data":"c9f85ca7495e6f784bdec88f084aee4b28770cde9d3fab3564b36cc4802ebd8f"} Jul 10 21:03:30.676865 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:30.676535 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz" Jul 10 21:03:30.695608 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:30.695557 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz" podStartSLOduration=1.9624989240000001 podStartE2EDuration="2.695541591s" totalImagesPullingTime="733.042667ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:03:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:03:30.69351609 +0000 UTC m=+1723.472097483" watchObservedRunningTime="2026-07-10 21:03:30.695541591 +0000 UTC m=+1723.474123025" Jul 10 21:03:34.103150 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:34.103112 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz"] Jul 10 21:03:34.103697 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:34.103325 2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz" podUID="58c0a40d-da09-45c4-83c8-49cba7d91b68" containerName="maas-api" containerID="cri-o://c9f85ca7495e6f784bdec88f084aee4b28770cde9d3fab3564b36cc4802ebd8f" gracePeriod=30 Jul 10 21:03:34.349624 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:34.349597 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz" Jul 10 21:03:34.476476 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:34.476443 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4gwz\" (UniqueName: \"kubernetes.io/projected/58c0a40d-da09-45c4-83c8-49cba7d91b68-kube-api-access-c4gwz\") pod \"58c0a40d-da09-45c4-83c8-49cba7d91b68\" (UID: \"58c0a40d-da09-45c4-83c8-49cba7d91b68\") " Jul 10 21:03:34.476648 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:34.476488 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58c0a40d-da09-45c4-83c8-49cba7d91b68-openshift-service-ca\") pod \"58c0a40d-da09-45c4-83c8-49cba7d91b68\" (UID: \"58c0a40d-da09-45c4-83c8-49cba7d91b68\") " Jul 10 21:03:34.476648 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:34.476553 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58c0a40d-da09-45c4-83c8-49cba7d91b68-maas-api-tls\") pod \"58c0a40d-da09-45c4-83c8-49cba7d91b68\" (UID: \"58c0a40d-da09-45c4-83c8-49cba7d91b68\") " Jul 10 21:03:34.476909 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:34.476880 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c0a40d-da09-45c4-83c8-49cba7d91b68-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "58c0a40d-da09-45c4-83c8-49cba7d91b68" (UID: "58c0a40d-da09-45c4-83c8-49cba7d91b68"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:03:34.478693 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:34.478664 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58c0a40d-da09-45c4-83c8-49cba7d91b68-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "58c0a40d-da09-45c4-83c8-49cba7d91b68" (UID: "58c0a40d-da09-45c4-83c8-49cba7d91b68"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:03:34.478771 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:34.478727 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c0a40d-da09-45c4-83c8-49cba7d91b68-kube-api-access-c4gwz" (OuterVolumeSpecName: "kube-api-access-c4gwz") pod "58c0a40d-da09-45c4-83c8-49cba7d91b68" (UID: "58c0a40d-da09-45c4-83c8-49cba7d91b68"). InnerVolumeSpecName "kube-api-access-c4gwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:03:34.577471 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:34.577432 2530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58c0a40d-da09-45c4-83c8-49cba7d91b68-openshift-service-ca\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:03:34.577471 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:34.577465 2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58c0a40d-da09-45c4-83c8-49cba7d91b68-maas-api-tls\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:03:34.577471 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:34.577474 2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4gwz\" (UniqueName: \"kubernetes.io/projected/58c0a40d-da09-45c4-83c8-49cba7d91b68-kube-api-access-c4gwz\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:03:34.696363 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:34.696329 2530 generic.go:356] "Generic (PLEG): container finished" podID="58c0a40d-da09-45c4-83c8-49cba7d91b68" containerID="c9f85ca7495e6f784bdec88f084aee4b28770cde9d3fab3564b36cc4802ebd8f" exitCode=0 Jul 10 21:03:34.696560 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:34.696372 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz" Jul 10 21:03:34.696560 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:34.696408 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz" event={"ID":"58c0a40d-da09-45c4-83c8-49cba7d91b68","Type":"ContainerDied","Data":"c9f85ca7495e6f784bdec88f084aee4b28770cde9d3fab3564b36cc4802ebd8f"} Jul 10 21:03:34.696560 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:34.696464 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz" event={"ID":"58c0a40d-da09-45c4-83c8-49cba7d91b68","Type":"ContainerDied","Data":"bac6bc3492cd3aca2337a44b45b2233da93d11b6dd3a1c3355f13d651d3da04f"} Jul 10 21:03:34.696560 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:34.696487 2530 scope.go:117] "RemoveContainer" containerID="c9f85ca7495e6f784bdec88f084aee4b28770cde9d3fab3564b36cc4802ebd8f" Jul 10 21:03:34.705031 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:34.705014 2530 scope.go:117] "RemoveContainer" containerID="c9f85ca7495e6f784bdec88f084aee4b28770cde9d3fab3564b36cc4802ebd8f" Jul 10 21:03:34.705315 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:03:34.705294 2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9f85ca7495e6f784bdec88f084aee4b28770cde9d3fab3564b36cc4802ebd8f\": container with ID starting with c9f85ca7495e6f784bdec88f084aee4b28770cde9d3fab3564b36cc4802ebd8f not found: ID does not exist" containerID="c9f85ca7495e6f784bdec88f084aee4b28770cde9d3fab3564b36cc4802ebd8f" Jul 10 21:03:34.705371 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:34.705324 2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9f85ca7495e6f784bdec88f084aee4b28770cde9d3fab3564b36cc4802ebd8f"} err="failed to get container status \"c9f85ca7495e6f784bdec88f084aee4b28770cde9d3fab3564b36cc4802ebd8f\": rpc error: code = NotFound desc = could not find container \"c9f85ca7495e6f784bdec88f084aee4b28770cde9d3fab3564b36cc4802ebd8f\": container with ID starting with c9f85ca7495e6f784bdec88f084aee4b28770cde9d3fab3564b36cc4802ebd8f not found: ID does not exist" Jul 10 21:03:34.717089 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:34.717063 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz"] Jul 10 21:03:34.720966 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:34.720941 2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6882a6b6-84996b9b98-r89qz"] Jul 10 21:03:35.771996 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:35.771964 2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58c0a40d-da09-45c4-83c8-49cba7d91b68" path="/var/lib/kubelet/pods/58c0a40d-da09-45c4-83c8-49cba7d91b68/volumes" Jul 10 21:03:41.441866 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:41.441814 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km"] Jul 10 21:03:41.442243 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:41.442141 2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58c0a40d-da09-45c4-83c8-49cba7d91b68" containerName="maas-api" Jul 10 21:03:41.442243 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:41.442152 2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="58c0a40d-da09-45c4-83c8-49cba7d91b68" containerName="maas-api" Jul 10 21:03:41.442243 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:41.442214 2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="58c0a40d-da09-45c4-83c8-49cba7d91b68" containerName="maas-api" Jul 10 21:03:41.446766 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:41.446743 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km" Jul 10 21:03:41.449151 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:41.449123 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pm7gf\"" Jul 10 21:03:41.449404 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:41.449388 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2c8fd457-serving-cert\"" Jul 10 21:03:41.459031 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:41.459005 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km"] Jul 10 21:03:41.522462 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:41.522426 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016f907e-d397-4572-968e-a928d1698941-openshift-service-ca\") pod \"maas-api-e2e-ait-2c8fd457-69b987d4d-775km\" (UID: \"016f907e-d397-4572-968e-a928d1698941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km" Jul 10 21:03:41.522666 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:41.522475 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp94t\" (UniqueName: \"kubernetes.io/projected/016f907e-d397-4572-968e-a928d1698941-kube-api-access-xp94t\") pod \"maas-api-e2e-ait-2c8fd457-69b987d4d-775km\" (UID: \"016f907e-d397-4572-968e-a928d1698941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km" Jul 10 21:03:41.522666 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:41.522519 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016f907e-d397-4572-968e-a928d1698941-maas-api-tls\") pod \"maas-api-e2e-ait-2c8fd457-69b987d4d-775km\" (UID: \"016f907e-d397-4572-968e-a928d1698941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km" Jul 10 21:03:41.623484 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:41.623445 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp94t\" (UniqueName: \"kubernetes.io/projected/016f907e-d397-4572-968e-a928d1698941-kube-api-access-xp94t\") pod \"maas-api-e2e-ait-2c8fd457-69b987d4d-775km\" (UID: \"016f907e-d397-4572-968e-a928d1698941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km" Jul 10 21:03:41.623691 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:41.623498 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016f907e-d397-4572-968e-a928d1698941-maas-api-tls\") pod \"maas-api-e2e-ait-2c8fd457-69b987d4d-775km\" (UID: \"016f907e-d397-4572-968e-a928d1698941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km" Jul 10 21:03:41.623691 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:03:41.623592 2530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-serving-cert: secret "maas-api-e2e-ait-2c8fd457-serving-cert" not found Jul 10 21:03:41.623691 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:03:41.623654 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/016f907e-d397-4572-968e-a928d1698941-maas-api-tls podName:016f907e-d397-4572-968e-a928d1698941 nodeName:}" failed. No retries permitted until 2026-07-10 21:03:42.123637854 +0000 UTC m=+1734.902219226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/016f907e-d397-4572-968e-a928d1698941-maas-api-tls") pod "maas-api-e2e-ait-2c8fd457-69b987d4d-775km" (UID: "016f907e-d397-4572-968e-a928d1698941") : secret "maas-api-e2e-ait-2c8fd457-serving-cert" not found Jul 10 21:03:41.623691 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:41.623595 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016f907e-d397-4572-968e-a928d1698941-openshift-service-ca\") pod \"maas-api-e2e-ait-2c8fd457-69b987d4d-775km\" (UID: \"016f907e-d397-4572-968e-a928d1698941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km" Jul 10 21:03:41.624318 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:41.624295 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016f907e-d397-4572-968e-a928d1698941-openshift-service-ca\") pod \"maas-api-e2e-ait-2c8fd457-69b987d4d-775km\" (UID: \"016f907e-d397-4572-968e-a928d1698941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km" Jul 10 21:03:41.632733 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:41.632706 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp94t\" (UniqueName: \"kubernetes.io/projected/016f907e-d397-4572-968e-a928d1698941-kube-api-access-xp94t\") pod \"maas-api-e2e-ait-2c8fd457-69b987d4d-775km\" (UID: \"016f907e-d397-4572-968e-a928d1698941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km" Jul 10 21:03:42.128871 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:42.128808 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016f907e-d397-4572-968e-a928d1698941-maas-api-tls\") pod \"maas-api-e2e-ait-2c8fd457-69b987d4d-775km\" (UID: \"016f907e-d397-4572-968e-a928d1698941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km" Jul 10 21:03:42.131332 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:42.131306 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016f907e-d397-4572-968e-a928d1698941-maas-api-tls\") pod \"maas-api-e2e-ait-2c8fd457-69b987d4d-775km\" (UID: \"016f907e-d397-4572-968e-a928d1698941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km" Jul 10 21:03:42.356753 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:42.356710 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km" Jul 10 21:03:42.487327 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:42.487295 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km"] Jul 10 21:03:42.493756 ip-10-0-139-64 kubenswrapper[2530]: W0710 21:03:42.493705 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016f907e_d397_4572_968e_a928d1698941.slice/crio-529b2c9e0e5ace40dc85e8131be4e1422c0752dd30c76298b5da5a99c8da0900 WatchSource:0}: Error finding container 529b2c9e0e5ace40dc85e8131be4e1422c0752dd30c76298b5da5a99c8da0900: Status 404 returned error can't find the container with id 529b2c9e0e5ace40dc85e8131be4e1422c0752dd30c76298b5da5a99c8da0900 Jul 10 21:03:42.732707 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:42.732618 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km" event={"ID":"016f907e-d397-4572-968e-a928d1698941","Type":"ContainerStarted","Data":"529b2c9e0e5ace40dc85e8131be4e1422c0752dd30c76298b5da5a99c8da0900"} Jul 10 21:03:43.744151 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:43.744111 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km" event={"ID":"016f907e-d397-4572-968e-a928d1698941","Type":"ContainerStarted","Data":"768443b0abac9183165887a2f7bf51386d519a757fd332a20abd28903cb82dc3"} Jul 10 21:03:43.744552 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:43.744271 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km" Jul 10 21:03:43.764341 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:43.764290 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km" podStartSLOduration=2.3831834880000002 podStartE2EDuration="2.764274934s" totalImagesPullingTime="381.091446ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:03:43.762277901 +0000 UTC m=+1736.540859307" watchObservedRunningTime="2026-07-10 21:03:43.764274934 +0000 UTC m=+1736.542856385" Jul 10 21:03:46.594963 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:46.594928 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km"] Jul 10 21:03:46.595406 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:46.595120 2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km" podUID="016f907e-d397-4572-968e-a928d1698941" containerName="maas-api" containerID="cri-o://768443b0abac9183165887a2f7bf51386d519a757fd332a20abd28903cb82dc3" gracePeriod=30 Jul 10 21:03:46.758971 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:46.758941 2530 generic.go:356] "Generic (PLEG): container finished" podID="016f907e-d397-4572-968e-a928d1698941" containerID="768443b0abac9183165887a2f7bf51386d519a757fd332a20abd28903cb82dc3" exitCode=0 Jul 10 21:03:46.759131 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:46.759013 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km" event={"ID":"016f907e-d397-4572-968e-a928d1698941","Type":"ContainerDied","Data":"768443b0abac9183165887a2f7bf51386d519a757fd332a20abd28903cb82dc3"} Jul 10 21:03:46.841635 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:46.841607 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km" Jul 10 21:03:46.872761 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:46.872686 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016f907e-d397-4572-968e-a928d1698941-openshift-service-ca\") pod \"016f907e-d397-4572-968e-a928d1698941\" (UID: \"016f907e-d397-4572-968e-a928d1698941\") " Jul 10 21:03:46.872761 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:46.872732 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016f907e-d397-4572-968e-a928d1698941-maas-api-tls\") pod \"016f907e-d397-4572-968e-a928d1698941\" (UID: \"016f907e-d397-4572-968e-a928d1698941\") " Jul 10 21:03:46.872968 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:46.872761 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp94t\" (UniqueName: \"kubernetes.io/projected/016f907e-d397-4572-968e-a928d1698941-kube-api-access-xp94t\") pod \"016f907e-d397-4572-968e-a928d1698941\" (UID: \"016f907e-d397-4572-968e-a928d1698941\") " Jul 10 21:03:46.873460 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:46.873221 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/016f907e-d397-4572-968e-a928d1698941-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "016f907e-d397-4572-968e-a928d1698941" (UID: "016f907e-d397-4572-968e-a928d1698941"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:03:46.875423 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:46.875386 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/016f907e-d397-4572-968e-a928d1698941-kube-api-access-xp94t" (OuterVolumeSpecName: "kube-api-access-xp94t") pod "016f907e-d397-4572-968e-a928d1698941" (UID: "016f907e-d397-4572-968e-a928d1698941"). InnerVolumeSpecName "kube-api-access-xp94t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:03:46.875759 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:46.875736 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/016f907e-d397-4572-968e-a928d1698941-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "016f907e-d397-4572-968e-a928d1698941" (UID: "016f907e-d397-4572-968e-a928d1698941"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:03:46.974198 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:46.974159 2530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016f907e-d397-4572-968e-a928d1698941-openshift-service-ca\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:03:46.974198 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:46.974192 2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016f907e-d397-4572-968e-a928d1698941-maas-api-tls\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:03:46.974198 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:46.974205 2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp94t\" (UniqueName: \"kubernetes.io/projected/016f907e-d397-4572-968e-a928d1698941-kube-api-access-xp94t\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:03:47.764079 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:47.764046 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km" Jul 10 21:03:47.764532 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:47.764075 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km" event={"ID":"016f907e-d397-4572-968e-a928d1698941","Type":"ContainerDied","Data":"529b2c9e0e5ace40dc85e8131be4e1422c0752dd30c76298b5da5a99c8da0900"} Jul 10 21:03:47.764532 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:47.764138 2530 scope.go:117] "RemoveContainer" containerID="768443b0abac9183165887a2f7bf51386d519a757fd332a20abd28903cb82dc3" Jul 10 21:03:47.786685 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:47.786656 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km"] Jul 10 21:03:47.790335 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:47.790314 2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2c8fd457-69b987d4d-775km"] Jul 10 21:03:49.772132 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:49.772087 2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="016f907e-d397-4572-968e-a928d1698941" path="/var/lib/kubelet/pods/016f907e-d397-4572-968e-a928d1698941/volumes" Jul 10 21:03:59.544857 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:59.544810 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w"] Jul 10 21:03:59.545293 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:59.545170 2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="016f907e-d397-4572-968e-a928d1698941" containerName="maas-api" Jul 10 21:03:59.545293 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:59.545189 2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="016f907e-d397-4572-968e-a928d1698941" containerName="maas-api" Jul 10 21:03:59.545293 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:59.545257 2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="016f907e-d397-4572-968e-a928d1698941" containerName="maas-api" Jul 10 21:03:59.548360 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:59.548339 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w" Jul 10 21:03:59.551151 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:59.551118 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pm7gf\"" Jul 10 21:03:59.551287 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:59.551159 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-e9b5f3ba-serving-cert\"" Jul 10 21:03:59.559673 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:59.559647 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w"] Jul 10 21:03:59.687413 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:59.687358 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e248130-ff67-4162-95dd-536eb3b8cf73-openshift-service-ca\") pod \"maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w\" (UID: \"3e248130-ff67-4162-95dd-536eb3b8cf73\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w" Jul 10 21:03:59.687413 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:59.687415 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqztp\" (UniqueName: \"kubernetes.io/projected/3e248130-ff67-4162-95dd-536eb3b8cf73-kube-api-access-wqztp\") pod \"maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w\" (UID: \"3e248130-ff67-4162-95dd-536eb3b8cf73\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w" Jul 10 21:03:59.687628 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:59.687554 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e248130-ff67-4162-95dd-536eb3b8cf73-maas-api-tls\") pod \"maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w\" (UID: \"3e248130-ff67-4162-95dd-536eb3b8cf73\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w" Jul 10 21:03:59.788932 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:59.788899 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e248130-ff67-4162-95dd-536eb3b8cf73-openshift-service-ca\") pod \"maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w\" (UID: \"3e248130-ff67-4162-95dd-536eb3b8cf73\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w" Jul 10 21:03:59.788932 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:59.788936 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqztp\" (UniqueName: \"kubernetes.io/projected/3e248130-ff67-4162-95dd-536eb3b8cf73-kube-api-access-wqztp\") pod \"maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w\" (UID: \"3e248130-ff67-4162-95dd-536eb3b8cf73\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w" Jul 10 21:03:59.789150 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:59.789015 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e248130-ff67-4162-95dd-536eb3b8cf73-maas-api-tls\") pod \"maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w\" (UID: \"3e248130-ff67-4162-95dd-536eb3b8cf73\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w" Jul 10 21:03:59.789150 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:03:59.789119 2530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-serving-cert: secret "maas-api-e2e-derive-e9b5f3ba-serving-cert" not found Jul 10 21:03:59.789238 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:03:59.789200 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e248130-ff67-4162-95dd-536eb3b8cf73-maas-api-tls podName:3e248130-ff67-4162-95dd-536eb3b8cf73 nodeName:}" failed. No retries permitted until 2026-07-10 21:04:00.289173893 +0000 UTC m=+1753.067755265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3e248130-ff67-4162-95dd-536eb3b8cf73-maas-api-tls") pod "maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w" (UID: "3e248130-ff67-4162-95dd-536eb3b8cf73") : secret "maas-api-e2e-derive-e9b5f3ba-serving-cert" not found Jul 10 21:03:59.789527 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:59.789507 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e248130-ff67-4162-95dd-536eb3b8cf73-openshift-service-ca\") pod \"maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w\" (UID: \"3e248130-ff67-4162-95dd-536eb3b8cf73\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w" Jul 10 21:03:59.807176 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:03:59.807116 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqztp\" (UniqueName: \"kubernetes.io/projected/3e248130-ff67-4162-95dd-536eb3b8cf73-kube-api-access-wqztp\") pod \"maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w\" (UID: \"3e248130-ff67-4162-95dd-536eb3b8cf73\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w" Jul 10 21:04:00.294378 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:00.294342 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e248130-ff67-4162-95dd-536eb3b8cf73-maas-api-tls\") pod \"maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w\" (UID: \"3e248130-ff67-4162-95dd-536eb3b8cf73\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w" Jul 10 21:04:00.296894 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:00.296871 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e248130-ff67-4162-95dd-536eb3b8cf73-maas-api-tls\") pod \"maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w\" (UID: \"3e248130-ff67-4162-95dd-536eb3b8cf73\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w" Jul 10 21:04:00.459578 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:00.459535 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w" Jul 10 21:04:00.595814 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:00.595790 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w"] Jul 10 21:04:00.598081 ip-10-0-139-64 kubenswrapper[2530]: W0710 21:04:00.598056 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e248130_ff67_4162_95dd_536eb3b8cf73.slice/crio-245fdc8564158bb6152b6d8bbeac04a8f7ae5da4b7d9f1d128dea357e53e9162 WatchSource:0}: Error finding container 245fdc8564158bb6152b6d8bbeac04a8f7ae5da4b7d9f1d128dea357e53e9162: Status 404 returned error can't find the container with id 245fdc8564158bb6152b6d8bbeac04a8f7ae5da4b7d9f1d128dea357e53e9162 Jul 10 21:04:00.826091 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:00.826058 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w" event={"ID":"3e248130-ff67-4162-95dd-536eb3b8cf73","Type":"ContainerStarted","Data":"245fdc8564158bb6152b6d8bbeac04a8f7ae5da4b7d9f1d128dea357e53e9162"} Jul 10 21:04:01.831842 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:01.831793 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w" event={"ID":"3e248130-ff67-4162-95dd-536eb3b8cf73","Type":"ContainerStarted","Data":"65cd36c80d5e51e2c7aa0b996d2f6bbc7117f1be60925bf78c13d79d808d767e"} Jul 10 21:04:01.832217 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:01.831868 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w" Jul 10 21:04:01.850842 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:01.850779 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w" podStartSLOduration=2.483117624 podStartE2EDuration="2.850762603s" totalImagesPullingTime="367.644979ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:03:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:04:01.848915997 +0000 UTC m=+1754.627497390" watchObservedRunningTime="2026-07-10 21:04:01.850762603 +0000 UTC m=+1754.629344018" Jul 10 21:04:04.171419 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:04.171364 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w"] Jul 10 21:04:04.171892 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:04.171660 2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w" podUID="3e248130-ff67-4162-95dd-536eb3b8cf73" containerName="maas-api" containerID="cri-o://65cd36c80d5e51e2c7aa0b996d2f6bbc7117f1be60925bf78c13d79d808d767e" gracePeriod=30 Jul 10 21:04:04.419888 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:04.419858 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w" Jul 10 21:04:04.535245 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:04.535157 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e248130-ff67-4162-95dd-536eb3b8cf73-maas-api-tls\") pod \"3e248130-ff67-4162-95dd-536eb3b8cf73\" (UID: \"3e248130-ff67-4162-95dd-536eb3b8cf73\") " Jul 10 21:04:04.535245 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:04.535206 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqztp\" (UniqueName: \"kubernetes.io/projected/3e248130-ff67-4162-95dd-536eb3b8cf73-kube-api-access-wqztp\") pod \"3e248130-ff67-4162-95dd-536eb3b8cf73\" (UID: \"3e248130-ff67-4162-95dd-536eb3b8cf73\") " Jul 10 21:04:04.535433 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:04.535262 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e248130-ff67-4162-95dd-536eb3b8cf73-openshift-service-ca\") pod \"3e248130-ff67-4162-95dd-536eb3b8cf73\" (UID: \"3e248130-ff67-4162-95dd-536eb3b8cf73\") " Jul 10 21:04:04.535670 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:04.535647 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e248130-ff67-4162-95dd-536eb3b8cf73-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3e248130-ff67-4162-95dd-536eb3b8cf73" (UID: "3e248130-ff67-4162-95dd-536eb3b8cf73"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:04:04.537363 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:04.537338 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e248130-ff67-4162-95dd-536eb3b8cf73-kube-api-access-wqztp" (OuterVolumeSpecName: "kube-api-access-wqztp") pod "3e248130-ff67-4162-95dd-536eb3b8cf73" (UID: "3e248130-ff67-4162-95dd-536eb3b8cf73"). InnerVolumeSpecName "kube-api-access-wqztp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:04:04.537470 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:04.537394 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e248130-ff67-4162-95dd-536eb3b8cf73-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3e248130-ff67-4162-95dd-536eb3b8cf73" (UID: "3e248130-ff67-4162-95dd-536eb3b8cf73"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:04:04.636364 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:04.636328 2530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e248130-ff67-4162-95dd-536eb3b8cf73-openshift-service-ca\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:04:04.636364 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:04.636358 2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e248130-ff67-4162-95dd-536eb3b8cf73-maas-api-tls\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:04:04.636364 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:04.636370 2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqztp\" (UniqueName: \"kubernetes.io/projected/3e248130-ff67-4162-95dd-536eb3b8cf73-kube-api-access-wqztp\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:04:04.845996 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:04.845911 2530 generic.go:356] "Generic (PLEG): container finished" podID="3e248130-ff67-4162-95dd-536eb3b8cf73" containerID="65cd36c80d5e51e2c7aa0b996d2f6bbc7117f1be60925bf78c13d79d808d767e" exitCode=0 Jul 10 21:04:04.845996 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:04.845948 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w" event={"ID":"3e248130-ff67-4162-95dd-536eb3b8cf73","Type":"ContainerDied","Data":"65cd36c80d5e51e2c7aa0b996d2f6bbc7117f1be60925bf78c13d79d808d767e"} Jul 10 21:04:04.845996 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:04.845957 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w" Jul 10 21:04:04.845996 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:04.845990 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w" event={"ID":"3e248130-ff67-4162-95dd-536eb3b8cf73","Type":"ContainerDied","Data":"245fdc8564158bb6152b6d8bbeac04a8f7ae5da4b7d9f1d128dea357e53e9162"} Jul 10 21:04:04.846248 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:04.846011 2530 scope.go:117] "RemoveContainer" containerID="65cd36c80d5e51e2c7aa0b996d2f6bbc7117f1be60925bf78c13d79d808d767e" Jul 10 21:04:04.854496 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:04.854461 2530 scope.go:117] "RemoveContainer" containerID="65cd36c80d5e51e2c7aa0b996d2f6bbc7117f1be60925bf78c13d79d808d767e" Jul 10 21:04:04.854746 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:04:04.854729 2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65cd36c80d5e51e2c7aa0b996d2f6bbc7117f1be60925bf78c13d79d808d767e\": container with ID starting with 65cd36c80d5e51e2c7aa0b996d2f6bbc7117f1be60925bf78c13d79d808d767e not found: ID does not exist" containerID="65cd36c80d5e51e2c7aa0b996d2f6bbc7117f1be60925bf78c13d79d808d767e" Jul 10 21:04:04.854811 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:04.854753 2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65cd36c80d5e51e2c7aa0b996d2f6bbc7117f1be60925bf78c13d79d808d767e"} err="failed to get container status \"65cd36c80d5e51e2c7aa0b996d2f6bbc7117f1be60925bf78c13d79d808d767e\": rpc error: code = NotFound desc = could not find container \"65cd36c80d5e51e2c7aa0b996d2f6bbc7117f1be60925bf78c13d79d808d767e\": container with ID starting with 65cd36c80d5e51e2c7aa0b996d2f6bbc7117f1be60925bf78c13d79d808d767e not found: ID does not exist" Jul 10 21:04:04.867755 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:04.867729 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w"] Jul 10 21:04:04.872392 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:04.872372 2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e9b5f3ba-7ccb7c75f-h7q4w"] Jul 10 21:04:05.772466 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:05.772438 2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e248130-ff67-4162-95dd-536eb3b8cf73" path="/var/lib/kubelet/pods/3e248130-ff67-4162-95dd-536eb3b8cf73/volumes" Jul 10 21:04:47.780665 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:47.780639 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n7kg6_a0fc144e-a4ca-4f83-9126-9366a781f0ad/ovn-acl-logging/0.log" Jul 10 21:04:47.784554 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:04:47.784533 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n7kg6_a0fc144e-a4ca-4f83-9126-9366a781f0ad/ovn-acl-logging/0.log" Jul 10 21:05:48.837791 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:48.837759 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt"] Jul 10 21:05:48.838363 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:48.838088 2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e248130-ff67-4162-95dd-536eb3b8cf73" containerName="maas-api" Jul 10 21:05:48.838363 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:48.838099 2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e248130-ff67-4162-95dd-536eb3b8cf73" containerName="maas-api" Jul 10 21:05:48.838363 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:48.838153 2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e248130-ff67-4162-95dd-536eb3b8cf73" containerName="maas-api" Jul 10 21:05:48.841070 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:48.841050 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt" Jul 10 21:05:48.843239 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:48.843210 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-76673b9a-serving-cert\"" Jul 10 21:05:48.843362 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:48.843288 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pm7gf\"" Jul 10 21:05:48.851982 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:48.851957 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt"] Jul 10 21:05:48.944756 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:48.944722 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32522df0-0ada-4c01-a538-74ca87f43da1-openshift-service-ca\") pod \"maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt\" (UID: \"32522df0-0ada-4c01-a538-74ca87f43da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt" Jul 10 21:05:48.944946 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:48.944800 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk9kd\" (UniqueName: \"kubernetes.io/projected/32522df0-0ada-4c01-a538-74ca87f43da1-kube-api-access-lk9kd\") pod \"maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt\" (UID: \"32522df0-0ada-4c01-a538-74ca87f43da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt" Jul 10 21:05:48.944946 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:48.944881 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32522df0-0ada-4c01-a538-74ca87f43da1-maas-api-tls\") pod \"maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt\" (UID: \"32522df0-0ada-4c01-a538-74ca87f43da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt" Jul 10 21:05:49.046198 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:49.046164 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32522df0-0ada-4c01-a538-74ca87f43da1-openshift-service-ca\") pod \"maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt\" (UID: \"32522df0-0ada-4c01-a538-74ca87f43da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt" Jul 10 21:05:49.046397 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:49.046259 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk9kd\" (UniqueName: \"kubernetes.io/projected/32522df0-0ada-4c01-a538-74ca87f43da1-kube-api-access-lk9kd\") pod \"maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt\" (UID: \"32522df0-0ada-4c01-a538-74ca87f43da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt" Jul 10 21:05:49.046397 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:49.046280 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32522df0-0ada-4c01-a538-74ca87f43da1-maas-api-tls\") pod \"maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt\" (UID: \"32522df0-0ada-4c01-a538-74ca87f43da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt" Jul 10 21:05:49.046523 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:05:49.046407 2530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-serving-cert: secret "maas-api-e2e-mt-76673b9a-serving-cert" not found Jul 10 21:05:49.046523 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:05:49.046482 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32522df0-0ada-4c01-a538-74ca87f43da1-maas-api-tls podName:32522df0-0ada-4c01-a538-74ca87f43da1 nodeName:}" failed. No retries permitted until 2026-07-10 21:05:49.546461178 +0000 UTC m=+1862.325042550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/32522df0-0ada-4c01-a538-74ca87f43da1-maas-api-tls") pod "maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt" (UID: "32522df0-0ada-4c01-a538-74ca87f43da1") : secret "maas-api-e2e-mt-76673b9a-serving-cert" not found Jul 10 21:05:49.046813 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:49.046788 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32522df0-0ada-4c01-a538-74ca87f43da1-openshift-service-ca\") pod \"maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt\" (UID: \"32522df0-0ada-4c01-a538-74ca87f43da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt" Jul 10 21:05:49.055350 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:49.055313 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk9kd\" (UniqueName: \"kubernetes.io/projected/32522df0-0ada-4c01-a538-74ca87f43da1-kube-api-access-lk9kd\") pod \"maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt\" (UID: \"32522df0-0ada-4c01-a538-74ca87f43da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt" Jul 10 21:05:49.550210 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:49.550177 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32522df0-0ada-4c01-a538-74ca87f43da1-maas-api-tls\") pod \"maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt\" (UID: \"32522df0-0ada-4c01-a538-74ca87f43da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt" Jul 10 21:05:49.550374 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:05:49.550322 2530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-serving-cert: secret "maas-api-e2e-mt-76673b9a-serving-cert" not found Jul 10 21:05:49.550425 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:05:49.550383 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32522df0-0ada-4c01-a538-74ca87f43da1-maas-api-tls podName:32522df0-0ada-4c01-a538-74ca87f43da1 nodeName:}" failed. No retries permitted until 2026-07-10 21:05:50.550366548 +0000 UTC m=+1863.328947926 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/32522df0-0ada-4c01-a538-74ca87f43da1-maas-api-tls") pod "maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt" (UID: "32522df0-0ada-4c01-a538-74ca87f43da1") : secret "maas-api-e2e-mt-76673b9a-serving-cert" not found Jul 10 21:05:50.559174 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:50.559140 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32522df0-0ada-4c01-a538-74ca87f43da1-maas-api-tls\") pod \"maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt\" (UID: \"32522df0-0ada-4c01-a538-74ca87f43da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt" Jul 10 21:05:50.561649 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:50.561624 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32522df0-0ada-4c01-a538-74ca87f43da1-maas-api-tls\") pod \"maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt\" (UID: \"32522df0-0ada-4c01-a538-74ca87f43da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt" Jul 10 21:05:50.652786 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:50.652745 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt" Jul 10 21:05:50.781587 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:50.781433 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt"] Jul 10 21:05:50.784201 ip-10-0-139-64 kubenswrapper[2530]: W0710 21:05:50.784176 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32522df0_0ada_4c01_a538_74ca87f43da1.slice/crio-194657cf5c625b5f0bd6b12314235c3c23e3c3aa41653216d0f6a40f90e28307 WatchSource:0}: Error finding container 194657cf5c625b5f0bd6b12314235c3c23e3c3aa41653216d0f6a40f90e28307: Status 404 returned error can't find the container with id 194657cf5c625b5f0bd6b12314235c3c23e3c3aa41653216d0f6a40f90e28307 Jul 10 21:05:50.785699 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:50.785679 2530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 21:05:51.285515 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:51.285481 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt" event={"ID":"32522df0-0ada-4c01-a538-74ca87f43da1","Type":"ContainerStarted","Data":"194657cf5c625b5f0bd6b12314235c3c23e3c3aa41653216d0f6a40f90e28307"} Jul 10 21:05:52.291004 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:52.290968 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt" event={"ID":"32522df0-0ada-4c01-a538-74ca87f43da1","Type":"ContainerStarted","Data":"a4080d0d5f393b2647ae873dbe7c17ccce88c76840d4dd556e091764f9142c99"} Jul 10 21:05:52.291452 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:52.291032 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt" Jul 10 21:05:52.308991 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:52.308942 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt" podStartSLOduration=3.807333038 podStartE2EDuration="4.308926734s" totalImagesPullingTime="501.593696ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:05:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:05:52.307138665 +0000 UTC m=+1865.085720060" watchObservedRunningTime="2026-07-10 21:05:52.308926734 +0000 UTC m=+1865.087508128" Jul 10 21:05:58.301038 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:05:58.301011 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt" Jul 10 21:06:00.979240 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:00.979205 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt"] Jul 10 21:06:00.979633 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:00.979407 2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt" podUID="32522df0-0ada-4c01-a538-74ca87f43da1" containerName="maas-api" containerID="cri-o://a4080d0d5f393b2647ae873dbe7c17ccce88c76840d4dd556e091764f9142c99" gracePeriod=30 Jul 10 21:06:01.220675 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:01.220652 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt" Jul 10 21:06:01.244282 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:01.244216 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lk9kd\" (UniqueName: \"kubernetes.io/projected/32522df0-0ada-4c01-a538-74ca87f43da1-kube-api-access-lk9kd\") pod \"32522df0-0ada-4c01-a538-74ca87f43da1\" (UID: \"32522df0-0ada-4c01-a538-74ca87f43da1\") " Jul 10 21:06:01.244282 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:01.244262 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32522df0-0ada-4c01-a538-74ca87f43da1-openshift-service-ca\") pod \"32522df0-0ada-4c01-a538-74ca87f43da1\" (UID: \"32522df0-0ada-4c01-a538-74ca87f43da1\") " Jul 10 21:06:01.244484 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:01.244307 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32522df0-0ada-4c01-a538-74ca87f43da1-maas-api-tls\") pod \"32522df0-0ada-4c01-a538-74ca87f43da1\" (UID: \"32522df0-0ada-4c01-a538-74ca87f43da1\") " Jul 10 21:06:01.244632 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:01.244606 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32522df0-0ada-4c01-a538-74ca87f43da1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "32522df0-0ada-4c01-a538-74ca87f43da1" (UID: "32522df0-0ada-4c01-a538-74ca87f43da1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:06:01.246429 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:01.246402 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32522df0-0ada-4c01-a538-74ca87f43da1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "32522df0-0ada-4c01-a538-74ca87f43da1" (UID: "32522df0-0ada-4c01-a538-74ca87f43da1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:06:01.246523 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:01.246438 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32522df0-0ada-4c01-a538-74ca87f43da1-kube-api-access-lk9kd" (OuterVolumeSpecName: "kube-api-access-lk9kd") pod "32522df0-0ada-4c01-a538-74ca87f43da1" (UID: "32522df0-0ada-4c01-a538-74ca87f43da1"). InnerVolumeSpecName "kube-api-access-lk9kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:06:01.330753 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:01.330719 2530 generic.go:356] "Generic (PLEG): container finished" podID="32522df0-0ada-4c01-a538-74ca87f43da1" containerID="a4080d0d5f393b2647ae873dbe7c17ccce88c76840d4dd556e091764f9142c99" exitCode=0 Jul 10 21:06:01.330946 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:01.330761 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt" Jul 10 21:06:01.330946 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:01.330803 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt" event={"ID":"32522df0-0ada-4c01-a538-74ca87f43da1","Type":"ContainerDied","Data":"a4080d0d5f393b2647ae873dbe7c17ccce88c76840d4dd556e091764f9142c99"} Jul 10 21:06:01.330946 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:01.330868 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt" event={"ID":"32522df0-0ada-4c01-a538-74ca87f43da1","Type":"ContainerDied","Data":"194657cf5c625b5f0bd6b12314235c3c23e3c3aa41653216d0f6a40f90e28307"} Jul 10 21:06:01.330946 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:01.330895 2530 scope.go:117] "RemoveContainer" containerID="a4080d0d5f393b2647ae873dbe7c17ccce88c76840d4dd556e091764f9142c99" Jul 10 21:06:01.339673 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:01.339654 2530 scope.go:117] "RemoveContainer" containerID="a4080d0d5f393b2647ae873dbe7c17ccce88c76840d4dd556e091764f9142c99" Jul 10 21:06:01.339961 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:06:01.339940 2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4080d0d5f393b2647ae873dbe7c17ccce88c76840d4dd556e091764f9142c99\": container with ID starting with a4080d0d5f393b2647ae873dbe7c17ccce88c76840d4dd556e091764f9142c99 not found: ID does not exist" containerID="a4080d0d5f393b2647ae873dbe7c17ccce88c76840d4dd556e091764f9142c99" Jul 10 21:06:01.340025 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:01.339969 2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4080d0d5f393b2647ae873dbe7c17ccce88c76840d4dd556e091764f9142c99"} err="failed to get container status \"a4080d0d5f393b2647ae873dbe7c17ccce88c76840d4dd556e091764f9142c99\": rpc error: code = NotFound desc = could not find container \"a4080d0d5f393b2647ae873dbe7c17ccce88c76840d4dd556e091764f9142c99\": container with ID starting with a4080d0d5f393b2647ae873dbe7c17ccce88c76840d4dd556e091764f9142c99 not found: ID does not exist" Jul 10 21:06:01.344984 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:01.344968 2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32522df0-0ada-4c01-a538-74ca87f43da1-maas-api-tls\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:06:01.345045 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:01.344986 2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk9kd\" (UniqueName: \"kubernetes.io/projected/32522df0-0ada-4c01-a538-74ca87f43da1-kube-api-access-lk9kd\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:06:01.345045 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:01.344996 2530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32522df0-0ada-4c01-a538-74ca87f43da1-openshift-service-ca\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:06:01.350757 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:01.350735 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt"] Jul 10 21:06:01.356414 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:01.356391 2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76673b9a-697df8d96f-dqxmt"] Jul 10 21:06:01.772187 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:01.772148 2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32522df0-0ada-4c01-a538-74ca87f43da1" path="/var/lib/kubelet/pods/32522df0-0ada-4c01-a538-74ca87f43da1/volumes" Jul 10 21:06:30.149087 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:30.149053 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm"] Jul 10 21:06:30.151550 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:30.149385 2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32522df0-0ada-4c01-a538-74ca87f43da1" containerName="maas-api" Jul 10 21:06:30.151550 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:30.149396 2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="32522df0-0ada-4c01-a538-74ca87f43da1" containerName="maas-api" Jul 10 21:06:30.151550 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:30.149448 2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="32522df0-0ada-4c01-a538-74ca87f43da1" containerName="maas-api" Jul 10 21:06:30.152556 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:30.152540 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm" Jul 10 21:06:30.155073 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:30.155051 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8d3029d7-serving-cert\"" Jul 10 21:06:30.155197 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:30.155111 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pm7gf\"" Jul 10 21:06:30.167085 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:30.167064 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm"] Jul 10 21:06:30.290312 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:30.290278 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/352cfab0-8200-4e88-91a0-3c6b4aea76be-openshift-service-ca\") pod \"maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm\" (UID: \"352cfab0-8200-4e88-91a0-3c6b4aea76be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm" Jul 10 21:06:30.290492 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:30.290330 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j4qf\" (UniqueName: \"kubernetes.io/projected/352cfab0-8200-4e88-91a0-3c6b4aea76be-kube-api-access-5j4qf\") pod \"maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm\" (UID: \"352cfab0-8200-4e88-91a0-3c6b4aea76be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm" Jul 10 21:06:30.290492 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:30.290385 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/352cfab0-8200-4e88-91a0-3c6b4aea76be-maas-api-tls\") pod \"maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm\" (UID: \"352cfab0-8200-4e88-91a0-3c6b4aea76be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm" Jul 10 21:06:30.391700 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:30.391664 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/352cfab0-8200-4e88-91a0-3c6b4aea76be-openshift-service-ca\") pod \"maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm\" (UID: \"352cfab0-8200-4e88-91a0-3c6b4aea76be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm" Jul 10 21:06:30.391700 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:30.391706 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j4qf\" (UniqueName: \"kubernetes.io/projected/352cfab0-8200-4e88-91a0-3c6b4aea76be-kube-api-access-5j4qf\") pod \"maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm\" (UID: \"352cfab0-8200-4e88-91a0-3c6b4aea76be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm" Jul 10 21:06:30.391938 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:30.391722 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/352cfab0-8200-4e88-91a0-3c6b4aea76be-maas-api-tls\") pod \"maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm\" (UID: \"352cfab0-8200-4e88-91a0-3c6b4aea76be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm" Jul 10 21:06:30.392364 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:30.392343 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/352cfab0-8200-4e88-91a0-3c6b4aea76be-openshift-service-ca\") pod \"maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm\" (UID: \"352cfab0-8200-4e88-91a0-3c6b4aea76be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm" Jul 10 21:06:30.394223 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:30.394205 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/352cfab0-8200-4e88-91a0-3c6b4aea76be-maas-api-tls\") pod \"maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm\" (UID: \"352cfab0-8200-4e88-91a0-3c6b4aea76be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm" Jul 10 21:06:30.405466 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:30.405407 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j4qf\" (UniqueName: \"kubernetes.io/projected/352cfab0-8200-4e88-91a0-3c6b4aea76be-kube-api-access-5j4qf\") pod \"maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm\" (UID: \"352cfab0-8200-4e88-91a0-3c6b4aea76be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm" Jul 10 21:06:30.462276 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:30.462247 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm" Jul 10 21:06:30.593353 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:30.593330 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm"] Jul 10 21:06:30.595867 ip-10-0-139-64 kubenswrapper[2530]: W0710 21:06:30.595812 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod352cfab0_8200_4e88_91a0_3c6b4aea76be.slice/crio-c1545e1bc64c60887c77275a2f8adf3abcfb5e382bd27309abba42a2a54de29d WatchSource:0}: Error finding container c1545e1bc64c60887c77275a2f8adf3abcfb5e382bd27309abba42a2a54de29d: Status 404 returned error can't find the container with id c1545e1bc64c60887c77275a2f8adf3abcfb5e382bd27309abba42a2a54de29d Jul 10 21:06:31.463547 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:31.463516 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm" event={"ID":"352cfab0-8200-4e88-91a0-3c6b4aea76be","Type":"ContainerStarted","Data":"c1545e1bc64c60887c77275a2f8adf3abcfb5e382bd27309abba42a2a54de29d"} Jul 10 21:06:32.469475 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:32.469436 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm" event={"ID":"352cfab0-8200-4e88-91a0-3c6b4aea76be","Type":"ContainerStarted","Data":"5ae65dd0fdded456b222625679aa38c5167a98396c07863078d77d2453bc9f5c"} Jul 10 21:06:32.469900 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:32.469544 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm" Jul 10 21:06:32.487445 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:32.487395 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm" podStartSLOduration=1.647624523 podStartE2EDuration="2.48737812s" totalImagesPullingTime="839.753597ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:06:32.485603966 +0000 UTC m=+1905.264185373" watchObservedRunningTime="2026-07-10 21:06:32.48737812 +0000 UTC m=+1905.265959514" Jul 10 21:06:38.478981 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:06:38.478949 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm" Jul 10 21:07:17.393859 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:07:17.393734 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm"] Jul 10 21:07:17.394458 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:07:17.394035 2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm" podUID="352cfab0-8200-4e88-91a0-3c6b4aea76be" containerName="maas-api" containerID="cri-o://5ae65dd0fdded456b222625679aa38c5167a98396c07863078d77d2453bc9f5c" gracePeriod=30 Jul 10 21:07:17.631501 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:07:17.631476 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm" Jul 10 21:07:17.662543 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:07:17.662467 2530 generic.go:356] "Generic (PLEG): container finished" podID="352cfab0-8200-4e88-91a0-3c6b4aea76be" containerID="5ae65dd0fdded456b222625679aa38c5167a98396c07863078d77d2453bc9f5c" exitCode=0 Jul 10 21:07:17.662543 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:07:17.662529 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm" Jul 10 21:07:17.662697 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:07:17.662554 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm" event={"ID":"352cfab0-8200-4e88-91a0-3c6b4aea76be","Type":"ContainerDied","Data":"5ae65dd0fdded456b222625679aa38c5167a98396c07863078d77d2453bc9f5c"} Jul 10 21:07:17.662697 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:07:17.662600 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm" event={"ID":"352cfab0-8200-4e88-91a0-3c6b4aea76be","Type":"ContainerDied","Data":"c1545e1bc64c60887c77275a2f8adf3abcfb5e382bd27309abba42a2a54de29d"} Jul 10 21:07:17.662697 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:07:17.662623 2530 scope.go:117] "RemoveContainer" containerID="5ae65dd0fdded456b222625679aa38c5167a98396c07863078d77d2453bc9f5c" Jul 10 21:07:17.670755 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:07:17.670726 2530 scope.go:117] "RemoveContainer" containerID="5ae65dd0fdded456b222625679aa38c5167a98396c07863078d77d2453bc9f5c" Jul 10 21:07:17.671047 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:07:17.671025 2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ae65dd0fdded456b222625679aa38c5167a98396c07863078d77d2453bc9f5c\": container with ID starting with 5ae65dd0fdded456b222625679aa38c5167a98396c07863078d77d2453bc9f5c not found: ID does not exist" containerID="5ae65dd0fdded456b222625679aa38c5167a98396c07863078d77d2453bc9f5c" Jul 10 21:07:17.671134 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:07:17.671055 2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ae65dd0fdded456b222625679aa38c5167a98396c07863078d77d2453bc9f5c"} err="failed to get container status \"5ae65dd0fdded456b222625679aa38c5167a98396c07863078d77d2453bc9f5c\": rpc error: code = NotFound desc = could not find container \"5ae65dd0fdded456b222625679aa38c5167a98396c07863078d77d2453bc9f5c\": container with ID starting with 5ae65dd0fdded456b222625679aa38c5167a98396c07863078d77d2453bc9f5c not found: ID does not exist" Jul 10 21:07:17.696242 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:07:17.696211 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/352cfab0-8200-4e88-91a0-3c6b4aea76be-openshift-service-ca\") pod \"352cfab0-8200-4e88-91a0-3c6b4aea76be\" (UID: \"352cfab0-8200-4e88-91a0-3c6b4aea76be\") " Jul 10 21:07:17.696353 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:07:17.696310 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/352cfab0-8200-4e88-91a0-3c6b4aea76be-maas-api-tls\") pod \"352cfab0-8200-4e88-91a0-3c6b4aea76be\" (UID: \"352cfab0-8200-4e88-91a0-3c6b4aea76be\") " Jul 10 21:07:17.696353 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:07:17.696349 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5j4qf\" (UniqueName: \"kubernetes.io/projected/352cfab0-8200-4e88-91a0-3c6b4aea76be-kube-api-access-5j4qf\") pod \"352cfab0-8200-4e88-91a0-3c6b4aea76be\" (UID: \"352cfab0-8200-4e88-91a0-3c6b4aea76be\") " Jul 10 21:07:17.696550 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:07:17.696530 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/352cfab0-8200-4e88-91a0-3c6b4aea76be-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "352cfab0-8200-4e88-91a0-3c6b4aea76be" (UID: "352cfab0-8200-4e88-91a0-3c6b4aea76be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:07:17.698241 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:07:17.698219 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/352cfab0-8200-4e88-91a0-3c6b4aea76be-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "352cfab0-8200-4e88-91a0-3c6b4aea76be" (UID: "352cfab0-8200-4e88-91a0-3c6b4aea76be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:07:17.698335 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:07:17.698315 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/352cfab0-8200-4e88-91a0-3c6b4aea76be-kube-api-access-5j4qf" (OuterVolumeSpecName: "kube-api-access-5j4qf") pod "352cfab0-8200-4e88-91a0-3c6b4aea76be" (UID: "352cfab0-8200-4e88-91a0-3c6b4aea76be"). InnerVolumeSpecName "kube-api-access-5j4qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:07:17.797260 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:07:17.797235 2530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/352cfab0-8200-4e88-91a0-3c6b4aea76be-openshift-service-ca\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:07:17.797260 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:07:17.797258 2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/352cfab0-8200-4e88-91a0-3c6b4aea76be-maas-api-tls\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:07:17.797395 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:07:17.797269 2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j4qf\" (UniqueName: \"kubernetes.io/projected/352cfab0-8200-4e88-91a0-3c6b4aea76be-kube-api-access-5j4qf\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:07:17.978377 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:07:17.978254 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm"] Jul 10 21:07:17.980156 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:07:17.980134 2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8d3029d7-6f7466c8b7-zwgxm"] Jul 10 21:07:19.772307 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:07:19.772273 2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="352cfab0-8200-4e88-91a0-3c6b4aea76be" path="/var/lib/kubelet/pods/352cfab0-8200-4e88-91a0-3c6b4aea76be/volumes" Jul 10 21:09:08.991705 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:08.991622 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6"] Jul 10 21:09:08.992209 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:08.991935 2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="352cfab0-8200-4e88-91a0-3c6b4aea76be" containerName="maas-api" Jul 10 21:09:08.992209 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:08.991945 2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="352cfab0-8200-4e88-91a0-3c6b4aea76be" containerName="maas-api" Jul 10 21:09:08.992209 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:08.992032 2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="352cfab0-8200-4e88-91a0-3c6b4aea76be" containerName="maas-api" Jul 10 21:09:09.000195 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:09.000168 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6" Jul 10 21:09:09.002638 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:09.002614 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7b3231-serving-cert\"" Jul 10 21:09:09.002777 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:09.002669 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pm7gf\"" Jul 10 21:09:09.006121 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:09.006058 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6"] Jul 10 21:09:09.164371 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:09.164338 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18324d5e-9481-45d2-a774-93a559bd9f8a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6\" (UID: \"18324d5e-9481-45d2-a774-93a559bd9f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6" Jul 10 21:09:09.164546 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:09.164387 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmcnj\" (UniqueName: \"kubernetes.io/projected/18324d5e-9481-45d2-a774-93a559bd9f8a-kube-api-access-hmcnj\") pod \"maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6\" (UID: \"18324d5e-9481-45d2-a774-93a559bd9f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6" Jul 10 21:09:09.164546 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:09.164518 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18324d5e-9481-45d2-a774-93a559bd9f8a-maas-api-tls\") pod \"maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6\" (UID: \"18324d5e-9481-45d2-a774-93a559bd9f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6" Jul 10 21:09:09.265762 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:09.265672 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmcnj\" (UniqueName: \"kubernetes.io/projected/18324d5e-9481-45d2-a774-93a559bd9f8a-kube-api-access-hmcnj\") pod \"maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6\" (UID: \"18324d5e-9481-45d2-a774-93a559bd9f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6" Jul 10 21:09:09.265762 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:09.265747 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18324d5e-9481-45d2-a774-93a559bd9f8a-maas-api-tls\") pod \"maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6\" (UID: \"18324d5e-9481-45d2-a774-93a559bd9f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6" Jul 10 21:09:09.265981 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:09.265774 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18324d5e-9481-45d2-a774-93a559bd9f8a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6\" (UID: \"18324d5e-9481-45d2-a774-93a559bd9f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6" Jul 10 21:09:09.265981 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:09:09.265918 2530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-serving-cert: secret "maas-api-e2e-inf-a-7b3231-serving-cert" not found Jul 10 21:09:09.266047 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:09:09.265994 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18324d5e-9481-45d2-a774-93a559bd9f8a-maas-api-tls podName:18324d5e-9481-45d2-a774-93a559bd9f8a nodeName:}" failed. No retries permitted until 2026-07-10 21:09:09.765977074 +0000 UTC m=+2062.544558465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/18324d5e-9481-45d2-a774-93a559bd9f8a-maas-api-tls") pod "maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6" (UID: "18324d5e-9481-45d2-a774-93a559bd9f8a") : secret "maas-api-e2e-inf-a-7b3231-serving-cert" not found Jul 10 21:09:09.266408 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:09.266378 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18324d5e-9481-45d2-a774-93a559bd9f8a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6\" (UID: \"18324d5e-9481-45d2-a774-93a559bd9f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6" Jul 10 21:09:09.274219 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:09.274197 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmcnj\" (UniqueName: \"kubernetes.io/projected/18324d5e-9481-45d2-a774-93a559bd9f8a-kube-api-access-hmcnj\") pod \"maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6\" (UID: \"18324d5e-9481-45d2-a774-93a559bd9f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6" Jul 10 21:09:09.773192 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:09.773152 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18324d5e-9481-45d2-a774-93a559bd9f8a-maas-api-tls\") pod \"maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6\" (UID: \"18324d5e-9481-45d2-a774-93a559bd9f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6" Jul 10 21:09:09.775568 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:09.775549 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18324d5e-9481-45d2-a774-93a559bd9f8a-maas-api-tls\") pod \"maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6\" (UID: \"18324d5e-9481-45d2-a774-93a559bd9f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6" Jul 10 21:09:09.911636 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:09.911592 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6" Jul 10 21:09:10.039516 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:10.039443 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6"] Jul 10 21:09:10.042664 ip-10-0-139-64 kubenswrapper[2530]: W0710 21:09:10.042637 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18324d5e_9481_45d2_a774_93a559bd9f8a.slice/crio-bc8d18c92b4571e9a92b86a4a8456843c1c70c873b457278939287752b22e97b WatchSource:0}: Error finding container bc8d18c92b4571e9a92b86a4a8456843c1c70c873b457278939287752b22e97b: Status 404 returned error can't find the container with id bc8d18c92b4571e9a92b86a4a8456843c1c70c873b457278939287752b22e97b Jul 10 21:09:10.130771 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:10.130737 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6" event={"ID":"18324d5e-9481-45d2-a774-93a559bd9f8a","Type":"ContainerStarted","Data":"bc8d18c92b4571e9a92b86a4a8456843c1c70c873b457278939287752b22e97b"} Jul 10 21:09:11.136137 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:11.136099 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6" event={"ID":"18324d5e-9481-45d2-a774-93a559bd9f8a","Type":"ContainerStarted","Data":"ecb3b99bc5d41a8cd5a34703f672138e1854544803380e46eb2a595c2911930a"} Jul 10 21:09:11.136487 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:11.136194 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6" Jul 10 21:09:11.155376 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:11.155332 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6" podStartSLOduration=2.37362807 podStartE2EDuration="3.155315833s" totalImagesPullingTime="781.687763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:09:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:09:11.154454358 +0000 UTC m=+2063.933035763" watchObservedRunningTime="2026-07-10 21:09:11.155315833 +0000 UTC m=+2063.933897225" Jul 10 21:09:17.066056 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:17.066021 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln"] Jul 10 21:09:17.069589 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:17.069567 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln" Jul 10 21:09:17.072013 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:17.071993 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-fc7575-serving-cert\"" Jul 10 21:09:17.082215 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:17.082151 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln"] Jul 10 21:09:17.126257 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:17.126220 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb6aece4-fcbb-4757-9956-dcbdaf78bed5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln\" (UID: \"eb6aece4-fcbb-4757-9956-dcbdaf78bed5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln" Jul 10 21:09:17.126436 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:17.126279 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb6aece4-fcbb-4757-9956-dcbdaf78bed5-maas-api-tls\") pod \"maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln\" (UID: \"eb6aece4-fcbb-4757-9956-dcbdaf78bed5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln" Jul 10 21:09:17.126436 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:17.126364 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn4tb\" (UniqueName: \"kubernetes.io/projected/eb6aece4-fcbb-4757-9956-dcbdaf78bed5-kube-api-access-zn4tb\") pod \"maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln\" (UID: \"eb6aece4-fcbb-4757-9956-dcbdaf78bed5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln" Jul 10 21:09:17.146665 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:17.146638 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6" Jul 10 21:09:17.227456 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:17.227421 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb6aece4-fcbb-4757-9956-dcbdaf78bed5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln\" (UID: \"eb6aece4-fcbb-4757-9956-dcbdaf78bed5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln" Jul 10 21:09:17.227641 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:17.227467 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb6aece4-fcbb-4757-9956-dcbdaf78bed5-maas-api-tls\") pod \"maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln\" (UID: \"eb6aece4-fcbb-4757-9956-dcbdaf78bed5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln" Jul 10 21:09:17.227641 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:17.227506 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn4tb\" (UniqueName: \"kubernetes.io/projected/eb6aece4-fcbb-4757-9956-dcbdaf78bed5-kube-api-access-zn4tb\") pod \"maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln\" (UID: \"eb6aece4-fcbb-4757-9956-dcbdaf78bed5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln" Jul 10 21:09:17.227641 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:09:17.227591 2530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-serving-cert: secret "maas-api-e2e-inf-b-fc7575-serving-cert" not found Jul 10 21:09:17.227813 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:09:17.227684 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb6aece4-fcbb-4757-9956-dcbdaf78bed5-maas-api-tls podName:eb6aece4-fcbb-4757-9956-dcbdaf78bed5 nodeName:}" failed. No retries permitted until 2026-07-10 21:09:17.72766477 +0000 UTC m=+2070.506246142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb6aece4-fcbb-4757-9956-dcbdaf78bed5-maas-api-tls") pod "maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln" (UID: "eb6aece4-fcbb-4757-9956-dcbdaf78bed5") : secret "maas-api-e2e-inf-b-fc7575-serving-cert" not found Jul 10 21:09:17.228151 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:17.228128 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb6aece4-fcbb-4757-9956-dcbdaf78bed5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln\" (UID: \"eb6aece4-fcbb-4757-9956-dcbdaf78bed5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln" Jul 10 21:09:17.236166 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:17.236143 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn4tb\" (UniqueName: \"kubernetes.io/projected/eb6aece4-fcbb-4757-9956-dcbdaf78bed5-kube-api-access-zn4tb\") pod \"maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln\" (UID: \"eb6aece4-fcbb-4757-9956-dcbdaf78bed5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln" Jul 10 21:09:17.732481 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:17.732448 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb6aece4-fcbb-4757-9956-dcbdaf78bed5-maas-api-tls\") pod \"maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln\" (UID: \"eb6aece4-fcbb-4757-9956-dcbdaf78bed5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln" Jul 10 21:09:17.734949 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:17.734921 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb6aece4-fcbb-4757-9956-dcbdaf78bed5-maas-api-tls\") pod \"maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln\" (UID: \"eb6aece4-fcbb-4757-9956-dcbdaf78bed5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln" Jul 10 21:09:17.983252 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:17.983163 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln" Jul 10 21:09:18.112814 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:18.112789 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln"] Jul 10 21:09:18.115197 ip-10-0-139-64 kubenswrapper[2530]: W0710 21:09:18.115170 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb6aece4_fcbb_4757_9956_dcbdaf78bed5.slice/crio-f39fcad2b2db81bf96366614ebdc8187701a906b4bbb742afa2c5d1a79126b00 WatchSource:0}: Error finding container f39fcad2b2db81bf96366614ebdc8187701a906b4bbb742afa2c5d1a79126b00: Status 404 returned error can't find the container with id f39fcad2b2db81bf96366614ebdc8187701a906b4bbb742afa2c5d1a79126b00 Jul 10 21:09:18.171793 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:18.171764 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln" event={"ID":"eb6aece4-fcbb-4757-9956-dcbdaf78bed5","Type":"ContainerStarted","Data":"f39fcad2b2db81bf96366614ebdc8187701a906b4bbb742afa2c5d1a79126b00"} Jul 10 21:09:19.177466 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:19.177429 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln" event={"ID":"eb6aece4-fcbb-4757-9956-dcbdaf78bed5","Type":"ContainerStarted","Data":"470e05bf05ed43eceda7d82b75d25693a8c827c6e527f27c5e2acb6765a3af7b"} Jul 10 21:09:19.177886 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:19.177535 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln" Jul 10 21:09:19.196556 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:19.196495 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln" podStartSLOduration=1.268566624 podStartE2EDuration="2.1964773s" totalImagesPullingTime="927.910676ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:09:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:09:19.195945761 +0000 UTC m=+2071.974527156" watchObservedRunningTime="2026-07-10 21:09:19.1964773 +0000 UTC m=+2071.975058694" Jul 10 21:09:25.187328 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:25.187297 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln" Jul 10 21:09:47.804710 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:47.804679 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n7kg6_a0fc144e-a4ca-4f83-9126-9366a781f0ad/ovn-acl-logging/0.log" Jul 10 21:09:47.810504 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:09:47.810483 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n7kg6_a0fc144e-a4ca-4f83-9126-9366a781f0ad/ovn-acl-logging/0.log" Jul 10 21:10:11.530882 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:11.530846 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6"] Jul 10 21:10:11.531321 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:11.531089 2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6" podUID="18324d5e-9481-45d2-a774-93a559bd9f8a" containerName="maas-api" containerID="cri-o://ecb3b99bc5d41a8cd5a34703f672138e1854544803380e46eb2a595c2911930a" gracePeriod=30 Jul 10 21:10:11.779312 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:11.779287 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6" Jul 10 21:10:11.792744 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:11.792678 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmcnj\" (UniqueName: \"kubernetes.io/projected/18324d5e-9481-45d2-a774-93a559bd9f8a-kube-api-access-hmcnj\") pod \"18324d5e-9481-45d2-a774-93a559bd9f8a\" (UID: \"18324d5e-9481-45d2-a774-93a559bd9f8a\") " Jul 10 21:10:11.792744 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:11.792735 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18324d5e-9481-45d2-a774-93a559bd9f8a-maas-api-tls\") pod \"18324d5e-9481-45d2-a774-93a559bd9f8a\" (UID: \"18324d5e-9481-45d2-a774-93a559bd9f8a\") " Jul 10 21:10:11.792930 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:11.792883 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18324d5e-9481-45d2-a774-93a559bd9f8a-openshift-service-ca\") pod \"18324d5e-9481-45d2-a774-93a559bd9f8a\" (UID: \"18324d5e-9481-45d2-a774-93a559bd9f8a\") " Jul 10 21:10:11.793242 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:11.793219 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18324d5e-9481-45d2-a774-93a559bd9f8a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "18324d5e-9481-45d2-a774-93a559bd9f8a" (UID: "18324d5e-9481-45d2-a774-93a559bd9f8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:10:11.794771 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:11.794749 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18324d5e-9481-45d2-a774-93a559bd9f8a-kube-api-access-hmcnj" (OuterVolumeSpecName: "kube-api-access-hmcnj") pod "18324d5e-9481-45d2-a774-93a559bd9f8a" (UID: "18324d5e-9481-45d2-a774-93a559bd9f8a"). InnerVolumeSpecName "kube-api-access-hmcnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:10:11.794954 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:11.794934 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18324d5e-9481-45d2-a774-93a559bd9f8a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "18324d5e-9481-45d2-a774-93a559bd9f8a" (UID: "18324d5e-9481-45d2-a774-93a559bd9f8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:10:11.894220 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:11.894172 2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmcnj\" (UniqueName: \"kubernetes.io/projected/18324d5e-9481-45d2-a774-93a559bd9f8a-kube-api-access-hmcnj\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:10:11.894220 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:11.894213 2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18324d5e-9481-45d2-a774-93a559bd9f8a-maas-api-tls\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:10:11.894220 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:11.894223 2530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18324d5e-9481-45d2-a774-93a559bd9f8a-openshift-service-ca\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:10:12.400541 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:12.400505 2530 generic.go:356] "Generic (PLEG): container finished" podID="18324d5e-9481-45d2-a774-93a559bd9f8a" containerID="ecb3b99bc5d41a8cd5a34703f672138e1854544803380e46eb2a595c2911930a" exitCode=0 Jul 10 21:10:12.400713 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:12.400555 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6" event={"ID":"18324d5e-9481-45d2-a774-93a559bd9f8a","Type":"ContainerDied","Data":"ecb3b99bc5d41a8cd5a34703f672138e1854544803380e46eb2a595c2911930a"} Jul 10 21:10:12.400713 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:12.400561 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6" Jul 10 21:10:12.400713 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:12.400635 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6" event={"ID":"18324d5e-9481-45d2-a774-93a559bd9f8a","Type":"ContainerDied","Data":"bc8d18c92b4571e9a92b86a4a8456843c1c70c873b457278939287752b22e97b"} Jul 10 21:10:12.400713 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:12.400659 2530 scope.go:117] "RemoveContainer" containerID="ecb3b99bc5d41a8cd5a34703f672138e1854544803380e46eb2a595c2911930a" Jul 10 21:10:12.409500 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:12.409477 2530 scope.go:117] "RemoveContainer" containerID="ecb3b99bc5d41a8cd5a34703f672138e1854544803380e46eb2a595c2911930a" Jul 10 21:10:12.409798 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:10:12.409776 2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecb3b99bc5d41a8cd5a34703f672138e1854544803380e46eb2a595c2911930a\": container with ID starting with ecb3b99bc5d41a8cd5a34703f672138e1854544803380e46eb2a595c2911930a not found: ID does not exist" containerID="ecb3b99bc5d41a8cd5a34703f672138e1854544803380e46eb2a595c2911930a" Jul 10 21:10:12.409870 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:12.409806 2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecb3b99bc5d41a8cd5a34703f672138e1854544803380e46eb2a595c2911930a"} err="failed to get container status \"ecb3b99bc5d41a8cd5a34703f672138e1854544803380e46eb2a595c2911930a\": rpc error: code = NotFound desc = could not find container \"ecb3b99bc5d41a8cd5a34703f672138e1854544803380e46eb2a595c2911930a\": container with ID starting with ecb3b99bc5d41a8cd5a34703f672138e1854544803380e46eb2a595c2911930a not found: ID does not exist" Jul 10 21:10:12.421632 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:12.421608 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6"] Jul 10 21:10:12.427825 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:12.427802 2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7b3231-649ff6bffb-w6bt6"] Jul 10 21:10:13.772810 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:13.772778 2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18324d5e-9481-45d2-a774-93a559bd9f8a" path="/var/lib/kubelet/pods/18324d5e-9481-45d2-a774-93a559bd9f8a/volumes" Jul 10 21:10:54.494331 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:54.494295 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln"] Jul 10 21:10:54.494864 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:54.494516 2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln" podUID="eb6aece4-fcbb-4757-9956-dcbdaf78bed5" containerName="maas-api" containerID="cri-o://470e05bf05ed43eceda7d82b75d25693a8c827c6e527f27c5e2acb6765a3af7b" gracePeriod=30 Jul 10 21:10:55.182241 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:55.182201 2530 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln" podUID="eb6aece4-fcbb-4757-9956-dcbdaf78bed5" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.34:8443/health\": dial tcp 10.133.0.34:8443: connect: connection refused" Jul 10 21:10:55.736074 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:55.736048 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln" Jul 10 21:10:55.782285 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:55.782246 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb6aece4-fcbb-4757-9956-dcbdaf78bed5-maas-api-tls\") pod \"eb6aece4-fcbb-4757-9956-dcbdaf78bed5\" (UID: \"eb6aece4-fcbb-4757-9956-dcbdaf78bed5\") " Jul 10 21:10:55.782452 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:55.782340 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb6aece4-fcbb-4757-9956-dcbdaf78bed5-openshift-service-ca\") pod \"eb6aece4-fcbb-4757-9956-dcbdaf78bed5\" (UID: \"eb6aece4-fcbb-4757-9956-dcbdaf78bed5\") " Jul 10 21:10:55.782452 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:55.782400 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn4tb\" (UniqueName: \"kubernetes.io/projected/eb6aece4-fcbb-4757-9956-dcbdaf78bed5-kube-api-access-zn4tb\") pod \"eb6aece4-fcbb-4757-9956-dcbdaf78bed5\" (UID: \"eb6aece4-fcbb-4757-9956-dcbdaf78bed5\") " Jul 10 21:10:55.782751 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:55.782712 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb6aece4-fcbb-4757-9956-dcbdaf78bed5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eb6aece4-fcbb-4757-9956-dcbdaf78bed5" (UID: "eb6aece4-fcbb-4757-9956-dcbdaf78bed5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:10:55.784458 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:55.784427 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb6aece4-fcbb-4757-9956-dcbdaf78bed5-kube-api-access-zn4tb" (OuterVolumeSpecName: "kube-api-access-zn4tb") pod "eb6aece4-fcbb-4757-9956-dcbdaf78bed5" (UID: "eb6aece4-fcbb-4757-9956-dcbdaf78bed5"). InnerVolumeSpecName "kube-api-access-zn4tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:10:55.784556 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:55.784494 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb6aece4-fcbb-4757-9956-dcbdaf78bed5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eb6aece4-fcbb-4757-9956-dcbdaf78bed5" (UID: "eb6aece4-fcbb-4757-9956-dcbdaf78bed5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:10:55.883965 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:55.883926 2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb6aece4-fcbb-4757-9956-dcbdaf78bed5-maas-api-tls\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:10:55.883965 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:55.883964 2530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb6aece4-fcbb-4757-9956-dcbdaf78bed5-openshift-service-ca\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:10:55.883965 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:55.883975 2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn4tb\" (UniqueName: \"kubernetes.io/projected/eb6aece4-fcbb-4757-9956-dcbdaf78bed5-kube-api-access-zn4tb\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:10:56.587792 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:56.587754 2530 generic.go:356] "Generic (PLEG): container finished" podID="eb6aece4-fcbb-4757-9956-dcbdaf78bed5" containerID="470e05bf05ed43eceda7d82b75d25693a8c827c6e527f27c5e2acb6765a3af7b" exitCode=0 Jul 10 21:10:56.587792 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:56.587796 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln" Jul 10 21:10:56.588034 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:56.587864 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln" event={"ID":"eb6aece4-fcbb-4757-9956-dcbdaf78bed5","Type":"ContainerDied","Data":"470e05bf05ed43eceda7d82b75d25693a8c827c6e527f27c5e2acb6765a3af7b"} Jul 10 21:10:56.588034 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:56.587911 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln" event={"ID":"eb6aece4-fcbb-4757-9956-dcbdaf78bed5","Type":"ContainerDied","Data":"f39fcad2b2db81bf96366614ebdc8187701a906b4bbb742afa2c5d1a79126b00"} Jul 10 21:10:56.588034 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:56.587936 2530 scope.go:117] "RemoveContainer" containerID="470e05bf05ed43eceda7d82b75d25693a8c827c6e527f27c5e2acb6765a3af7b" Jul 10 21:10:56.596386 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:56.596370 2530 scope.go:117] "RemoveContainer" containerID="470e05bf05ed43eceda7d82b75d25693a8c827c6e527f27c5e2acb6765a3af7b" Jul 10 21:10:56.596644 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:10:56.596626 2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"470e05bf05ed43eceda7d82b75d25693a8c827c6e527f27c5e2acb6765a3af7b\": container with ID starting with 470e05bf05ed43eceda7d82b75d25693a8c827c6e527f27c5e2acb6765a3af7b not found: ID does not exist" containerID="470e05bf05ed43eceda7d82b75d25693a8c827c6e527f27c5e2acb6765a3af7b" Jul 10 21:10:56.596711 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:56.596653 2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"470e05bf05ed43eceda7d82b75d25693a8c827c6e527f27c5e2acb6765a3af7b"} err="failed to get container status \"470e05bf05ed43eceda7d82b75d25693a8c827c6e527f27c5e2acb6765a3af7b\": rpc error: code = NotFound desc = could not find container \"470e05bf05ed43eceda7d82b75d25693a8c827c6e527f27c5e2acb6765a3af7b\": container with ID starting with 470e05bf05ed43eceda7d82b75d25693a8c827c6e527f27c5e2acb6765a3af7b not found: ID does not exist" Jul 10 21:10:56.608676 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:56.608647 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln"] Jul 10 21:10:56.616275 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:56.616253 2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fc7575-584d548fd8-xzvln"] Jul 10 21:10:57.771957 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:10:57.771925 2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb6aece4-fcbb-4757-9956-dcbdaf78bed5" path="/var/lib/kubelet/pods/eb6aece4-fcbb-4757-9956-dcbdaf78bed5/volumes" Jul 10 21:11:04.625820 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:04.625788 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25"] Jul 10 21:11:04.626295 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:04.626129 2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18324d5e-9481-45d2-a774-93a559bd9f8a" containerName="maas-api" Jul 10 21:11:04.626295 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:04.626141 2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="18324d5e-9481-45d2-a774-93a559bd9f8a" containerName="maas-api" Jul 10 21:11:04.626295 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:04.626155 2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb6aece4-fcbb-4757-9956-dcbdaf78bed5" containerName="maas-api" Jul 10 21:11:04.626295 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:04.626161 2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb6aece4-fcbb-4757-9956-dcbdaf78bed5" containerName="maas-api" Jul 10 21:11:04.626295 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:04.626231 2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb6aece4-fcbb-4757-9956-dcbdaf78bed5" containerName="maas-api" Jul 10 21:11:04.626295 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:04.626243 2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="18324d5e-9481-45d2-a774-93a559bd9f8a" containerName="maas-api" Jul 10 21:11:04.630979 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:04.630957 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25" Jul 10 21:11:04.636241 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:04.636220 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-8b36b2-serving-cert\"" Jul 10 21:11:04.637439 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:04.637418 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pm7gf\"" Jul 10 21:11:04.648699 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:04.648676 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25"] Jul 10 21:11:04.664320 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:04.664295 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be794ce1-bf62-447a-ae1e-a4612c5f115d-openshift-service-ca\") pod \"maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25\" (UID: \"be794ce1-bf62-447a-ae1e-a4612c5f115d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25" Jul 10 21:11:04.664470 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:04.664349 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fffzj\" (UniqueName: \"kubernetes.io/projected/be794ce1-bf62-447a-ae1e-a4612c5f115d-kube-api-access-fffzj\") pod \"maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25\" (UID: \"be794ce1-bf62-447a-ae1e-a4612c5f115d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25" Jul 10 21:11:04.664470 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:04.664419 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be794ce1-bf62-447a-ae1e-a4612c5f115d-maas-api-tls\") pod \"maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25\" (UID: \"be794ce1-bf62-447a-ae1e-a4612c5f115d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25" Jul 10 21:11:04.765015 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:04.764976 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fffzj\" (UniqueName: \"kubernetes.io/projected/be794ce1-bf62-447a-ae1e-a4612c5f115d-kube-api-access-fffzj\") pod \"maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25\" (UID: \"be794ce1-bf62-447a-ae1e-a4612c5f115d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25" Jul 10 21:11:04.765195 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:04.765024 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be794ce1-bf62-447a-ae1e-a4612c5f115d-maas-api-tls\") pod \"maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25\" (UID: \"be794ce1-bf62-447a-ae1e-a4612c5f115d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25" Jul 10 21:11:04.765195 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:04.765066 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be794ce1-bf62-447a-ae1e-a4612c5f115d-openshift-service-ca\") pod \"maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25\" (UID: \"be794ce1-bf62-447a-ae1e-a4612c5f115d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25" Jul 10 21:11:04.765329 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:11:04.765220 2530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-serving-cert: secret "maas-api-e2e-api-a-8b36b2-serving-cert" not found Jul 10 21:11:04.765329 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:11:04.765283 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be794ce1-bf62-447a-ae1e-a4612c5f115d-maas-api-tls podName:be794ce1-bf62-447a-ae1e-a4612c5f115d nodeName:}" failed. No retries permitted until 2026-07-10 21:11:05.265266998 +0000 UTC m=+2178.043848373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/be794ce1-bf62-447a-ae1e-a4612c5f115d-maas-api-tls") pod "maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25" (UID: "be794ce1-bf62-447a-ae1e-a4612c5f115d") : secret "maas-api-e2e-api-a-8b36b2-serving-cert" not found Jul 10 21:11:04.765666 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:04.765647 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be794ce1-bf62-447a-ae1e-a4612c5f115d-openshift-service-ca\") pod \"maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25\" (UID: \"be794ce1-bf62-447a-ae1e-a4612c5f115d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25" Jul 10 21:11:04.775702 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:04.775681 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fffzj\" (UniqueName: \"kubernetes.io/projected/be794ce1-bf62-447a-ae1e-a4612c5f115d-kube-api-access-fffzj\") pod \"maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25\" (UID: \"be794ce1-bf62-447a-ae1e-a4612c5f115d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25" Jul 10 21:11:05.270527 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:05.270488 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be794ce1-bf62-447a-ae1e-a4612c5f115d-maas-api-tls\") pod \"maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25\" (UID: \"be794ce1-bf62-447a-ae1e-a4612c5f115d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25" Jul 10 21:11:05.272875 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:05.272845 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be794ce1-bf62-447a-ae1e-a4612c5f115d-maas-api-tls\") pod \"maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25\" (UID: \"be794ce1-bf62-447a-ae1e-a4612c5f115d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25" Jul 10 21:11:05.541678 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:05.541578 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25" Jul 10 21:11:05.677648 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:05.677623 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25"] Jul 10 21:11:05.680082 ip-10-0-139-64 kubenswrapper[2530]: W0710 21:11:05.680047 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe794ce1_bf62_447a_ae1e_a4612c5f115d.slice/crio-48ad2e7769ac12c3a593abfcc4316a7d31df2afb7f402b722ea4f41b924d6a12 WatchSource:0}: Error finding container 48ad2e7769ac12c3a593abfcc4316a7d31df2afb7f402b722ea4f41b924d6a12: Status 404 returned error can't find the container with id 48ad2e7769ac12c3a593abfcc4316a7d31df2afb7f402b722ea4f41b924d6a12 Jul 10 21:11:05.681698 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:05.681677 2530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 21:11:06.635434 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:06.635397 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25" event={"ID":"be794ce1-bf62-447a-ae1e-a4612c5f115d","Type":"ContainerStarted","Data":"322a4d5c69c4648026c5edd4e6a8f041eb688ca8f9976b1524414892ab9d6ca7"} Jul 10 21:11:06.635612 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:06.635459 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25" event={"ID":"be794ce1-bf62-447a-ae1e-a4612c5f115d","Type":"ContainerStarted","Data":"48ad2e7769ac12c3a593abfcc4316a7d31df2afb7f402b722ea4f41b924d6a12"} Jul 10 21:11:06.635612 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:06.635509 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25" Jul 10 21:11:06.655200 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:06.655150 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25" podStartSLOduration=2.23998963 podStartE2EDuration="2.65513311s" totalImagesPullingTime="415.14348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:11:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:11:06.653302355 +0000 UTC m=+2179.431883748" watchObservedRunningTime="2026-07-10 21:11:06.65513311 +0000 UTC m=+2179.433714504" Jul 10 21:11:12.646822 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:12.646790 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25" Jul 10 21:11:12.751703 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:12.751673 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr"] Jul 10 21:11:12.755158 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:12.755131 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr" Jul 10 21:11:12.758477 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:12.758455 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0b11f5-serving-cert\"" Jul 10 21:11:12.766368 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:12.766345 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr"] Jul 10 21:11:12.831480 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:12.831443 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j9r4\" (UniqueName: \"kubernetes.io/projected/179f2c3a-8342-4e7b-b626-1de15c0bc374-kube-api-access-4j9r4\") pod \"maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr\" (UID: \"179f2c3a-8342-4e7b-b626-1de15c0bc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr" Jul 10 21:11:12.831642 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:12.831506 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/179f2c3a-8342-4e7b-b626-1de15c0bc374-openshift-service-ca\") pod \"maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr\" (UID: \"179f2c3a-8342-4e7b-b626-1de15c0bc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr" Jul 10 21:11:12.831642 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:12.831589 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/179f2c3a-8342-4e7b-b626-1de15c0bc374-maas-api-tls\") pod \"maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr\" (UID: \"179f2c3a-8342-4e7b-b626-1de15c0bc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr" Jul 10 21:11:12.933064 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:12.932957 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/179f2c3a-8342-4e7b-b626-1de15c0bc374-openshift-service-ca\") pod \"maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr\" (UID: \"179f2c3a-8342-4e7b-b626-1de15c0bc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr" Jul 10 21:11:12.933064 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:12.933016 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/179f2c3a-8342-4e7b-b626-1de15c0bc374-maas-api-tls\") pod \"maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr\" (UID: \"179f2c3a-8342-4e7b-b626-1de15c0bc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr" Jul 10 21:11:12.933304 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:12.933104 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j9r4\" (UniqueName: \"kubernetes.io/projected/179f2c3a-8342-4e7b-b626-1de15c0bc374-kube-api-access-4j9r4\") pod \"maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr\" (UID: \"179f2c3a-8342-4e7b-b626-1de15c0bc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr" Jul 10 21:11:12.933663 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:12.933632 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/179f2c3a-8342-4e7b-b626-1de15c0bc374-openshift-service-ca\") pod \"maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr\" (UID: \"179f2c3a-8342-4e7b-b626-1de15c0bc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr" Jul 10 21:11:12.935465 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:12.935447 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/179f2c3a-8342-4e7b-b626-1de15c0bc374-maas-api-tls\") pod \"maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr\" (UID: \"179f2c3a-8342-4e7b-b626-1de15c0bc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr" Jul 10 21:11:12.940664 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:12.940643 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j9r4\" (UniqueName: \"kubernetes.io/projected/179f2c3a-8342-4e7b-b626-1de15c0bc374-kube-api-access-4j9r4\") pod \"maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr\" (UID: \"179f2c3a-8342-4e7b-b626-1de15c0bc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr" Jul 10 21:11:13.066230 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:13.066187 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr" Jul 10 21:11:13.204715 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:13.204689 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr"] Jul 10 21:11:13.207581 ip-10-0-139-64 kubenswrapper[2530]: W0710 21:11:13.207548 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179f2c3a_8342_4e7b_b626_1de15c0bc374.slice/crio-02a3f5ec52060545667d6029669b720798a15d123b850d8fd0e1be5f45b4d2fa WatchSource:0}: Error finding container 02a3f5ec52060545667d6029669b720798a15d123b850d8fd0e1be5f45b4d2fa: Status 404 returned error can't find the container with id 02a3f5ec52060545667d6029669b720798a15d123b850d8fd0e1be5f45b4d2fa Jul 10 21:11:13.668126 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:13.668088 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr" event={"ID":"179f2c3a-8342-4e7b-b626-1de15c0bc374","Type":"ContainerStarted","Data":"af9b0addfe5425eba60bdad5f0de5a779431f44c3ca8a318e05bae451d33c56b"} Jul 10 21:11:13.668524 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:13.668143 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr" event={"ID":"179f2c3a-8342-4e7b-b626-1de15c0bc374","Type":"ContainerStarted","Data":"02a3f5ec52060545667d6029669b720798a15d123b850d8fd0e1be5f45b4d2fa"} Jul 10 21:11:13.668524 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:13.668213 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr" Jul 10 21:11:13.685400 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:13.685349 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr" podStartSLOduration=1.3501952099999999 podStartE2EDuration="1.685331856s" totalImagesPullingTime="335.136646ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:11:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:11:13.684520806 +0000 UTC m=+2186.463102200" watchObservedRunningTime="2026-07-10 21:11:13.685331856 +0000 UTC m=+2186.463913288" Jul 10 21:11:19.678742 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:19.678711 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr" Jul 10 21:11:25.897013 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:25.896970 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25"] Jul 10 21:11:25.897485 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:25.897249 2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25" podUID="be794ce1-bf62-447a-ae1e-a4612c5f115d" containerName="maas-api" containerID="cri-o://322a4d5c69c4648026c5edd4e6a8f041eb688ca8f9976b1524414892ab9d6ca7" gracePeriod=30 Jul 10 21:11:26.146763 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:26.146736 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25" Jul 10 21:11:26.244646 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:26.244556 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be794ce1-bf62-447a-ae1e-a4612c5f115d-openshift-service-ca\") pod \"be794ce1-bf62-447a-ae1e-a4612c5f115d\" (UID: \"be794ce1-bf62-447a-ae1e-a4612c5f115d\") " Jul 10 21:11:26.244646 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:26.244613 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be794ce1-bf62-447a-ae1e-a4612c5f115d-maas-api-tls\") pod \"be794ce1-bf62-447a-ae1e-a4612c5f115d\" (UID: \"be794ce1-bf62-447a-ae1e-a4612c5f115d\") " Jul 10 21:11:26.244897 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:26.244668 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fffzj\" (UniqueName: \"kubernetes.io/projected/be794ce1-bf62-447a-ae1e-a4612c5f115d-kube-api-access-fffzj\") pod \"be794ce1-bf62-447a-ae1e-a4612c5f115d\" (UID: \"be794ce1-bf62-447a-ae1e-a4612c5f115d\") " Jul 10 21:11:26.244978 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:26.244945 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be794ce1-bf62-447a-ae1e-a4612c5f115d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "be794ce1-bf62-447a-ae1e-a4612c5f115d" (UID: "be794ce1-bf62-447a-ae1e-a4612c5f115d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:11:26.246721 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:26.246692 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be794ce1-bf62-447a-ae1e-a4612c5f115d-kube-api-access-fffzj" (OuterVolumeSpecName: "kube-api-access-fffzj") pod "be794ce1-bf62-447a-ae1e-a4612c5f115d" (UID: "be794ce1-bf62-447a-ae1e-a4612c5f115d"). InnerVolumeSpecName "kube-api-access-fffzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:11:26.246721 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:26.246700 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be794ce1-bf62-447a-ae1e-a4612c5f115d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "be794ce1-bf62-447a-ae1e-a4612c5f115d" (UID: "be794ce1-bf62-447a-ae1e-a4612c5f115d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:11:26.345631 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:26.345595 2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fffzj\" (UniqueName: \"kubernetes.io/projected/be794ce1-bf62-447a-ae1e-a4612c5f115d-kube-api-access-fffzj\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:11:26.345631 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:26.345624 2530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be794ce1-bf62-447a-ae1e-a4612c5f115d-openshift-service-ca\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:11:26.345631 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:26.345636 2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be794ce1-bf62-447a-ae1e-a4612c5f115d-maas-api-tls\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:11:26.727407 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:26.727371 2530 generic.go:356] "Generic (PLEG): container finished" podID="be794ce1-bf62-447a-ae1e-a4612c5f115d" containerID="322a4d5c69c4648026c5edd4e6a8f041eb688ca8f9976b1524414892ab9d6ca7" exitCode=0 Jul 10 21:11:26.727599 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:26.727419 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25" Jul 10 21:11:26.727599 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:26.727420 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25" event={"ID":"be794ce1-bf62-447a-ae1e-a4612c5f115d","Type":"ContainerDied","Data":"322a4d5c69c4648026c5edd4e6a8f041eb688ca8f9976b1524414892ab9d6ca7"} Jul 10 21:11:26.727599 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:26.727562 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25" event={"ID":"be794ce1-bf62-447a-ae1e-a4612c5f115d","Type":"ContainerDied","Data":"48ad2e7769ac12c3a593abfcc4316a7d31df2afb7f402b722ea4f41b924d6a12"} Jul 10 21:11:26.727774 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:26.727599 2530 scope.go:117] "RemoveContainer" containerID="322a4d5c69c4648026c5edd4e6a8f041eb688ca8f9976b1524414892ab9d6ca7" Jul 10 21:11:26.736082 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:26.736065 2530 scope.go:117] "RemoveContainer" containerID="322a4d5c69c4648026c5edd4e6a8f041eb688ca8f9976b1524414892ab9d6ca7" Jul 10 21:11:26.736372 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:11:26.736352 2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"322a4d5c69c4648026c5edd4e6a8f041eb688ca8f9976b1524414892ab9d6ca7\": container with ID starting with 322a4d5c69c4648026c5edd4e6a8f041eb688ca8f9976b1524414892ab9d6ca7 not found: ID does not exist" containerID="322a4d5c69c4648026c5edd4e6a8f041eb688ca8f9976b1524414892ab9d6ca7" Jul 10 21:11:26.736416 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:26.736381 2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322a4d5c69c4648026c5edd4e6a8f041eb688ca8f9976b1524414892ab9d6ca7"} err="failed to get container status \"322a4d5c69c4648026c5edd4e6a8f041eb688ca8f9976b1524414892ab9d6ca7\": rpc error: code = NotFound desc = could not find container \"322a4d5c69c4648026c5edd4e6a8f041eb688ca8f9976b1524414892ab9d6ca7\": container with ID starting with 322a4d5c69c4648026c5edd4e6a8f041eb688ca8f9976b1524414892ab9d6ca7 not found: ID does not exist" Jul 10 21:11:26.747763 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:26.747733 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25"] Jul 10 21:11:26.753616 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:26.753592 2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8b36b2-6674c49cdb-26g25"] Jul 10 21:11:27.772167 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:27.772135 2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be794ce1-bf62-447a-ae1e-a4612c5f115d" path="/var/lib/kubelet/pods/be794ce1-bf62-447a-ae1e-a4612c5f115d/volumes" Jul 10 21:11:32.812626 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:32.812590 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr"] Jul 10 21:11:32.815336 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:32.812805 2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr" podUID="179f2c3a-8342-4e7b-b626-1de15c0bc374" containerName="maas-api" containerID="cri-o://af9b0addfe5425eba60bdad5f0de5a779431f44c3ca8a318e05bae451d33c56b" gracePeriod=30 Jul 10 21:11:33.064463 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:33.064403 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr" Jul 10 21:11:33.207737 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:33.207697 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/179f2c3a-8342-4e7b-b626-1de15c0bc374-maas-api-tls\") pod \"179f2c3a-8342-4e7b-b626-1de15c0bc374\" (UID: \"179f2c3a-8342-4e7b-b626-1de15c0bc374\") " Jul 10 21:11:33.207960 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:33.207752 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/179f2c3a-8342-4e7b-b626-1de15c0bc374-openshift-service-ca\") pod \"179f2c3a-8342-4e7b-b626-1de15c0bc374\" (UID: \"179f2c3a-8342-4e7b-b626-1de15c0bc374\") " Jul 10 21:11:33.207960 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:33.207824 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4j9r4\" (UniqueName: \"kubernetes.io/projected/179f2c3a-8342-4e7b-b626-1de15c0bc374-kube-api-access-4j9r4\") pod \"179f2c3a-8342-4e7b-b626-1de15c0bc374\" (UID: \"179f2c3a-8342-4e7b-b626-1de15c0bc374\") " Jul 10 21:11:33.208259 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:33.208232 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/179f2c3a-8342-4e7b-b626-1de15c0bc374-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "179f2c3a-8342-4e7b-b626-1de15c0bc374" (UID: "179f2c3a-8342-4e7b-b626-1de15c0bc374"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:11:33.209880 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:33.209853 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/179f2c3a-8342-4e7b-b626-1de15c0bc374-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "179f2c3a-8342-4e7b-b626-1de15c0bc374" (UID: "179f2c3a-8342-4e7b-b626-1de15c0bc374"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:11:33.209961 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:33.209912 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/179f2c3a-8342-4e7b-b626-1de15c0bc374-kube-api-access-4j9r4" (OuterVolumeSpecName: "kube-api-access-4j9r4") pod "179f2c3a-8342-4e7b-b626-1de15c0bc374" (UID: "179f2c3a-8342-4e7b-b626-1de15c0bc374"). InnerVolumeSpecName "kube-api-access-4j9r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:11:33.308675 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:33.308642 2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/179f2c3a-8342-4e7b-b626-1de15c0bc374-maas-api-tls\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:11:33.308675 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:33.308666 2530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/179f2c3a-8342-4e7b-b626-1de15c0bc374-openshift-service-ca\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:11:33.308675 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:33.308677 2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j9r4\" (UniqueName: \"kubernetes.io/projected/179f2c3a-8342-4e7b-b626-1de15c0bc374-kube-api-access-4j9r4\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:11:33.760415 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:33.760380 2530 generic.go:356] "Generic (PLEG): container finished" podID="179f2c3a-8342-4e7b-b626-1de15c0bc374" containerID="af9b0addfe5425eba60bdad5f0de5a779431f44c3ca8a318e05bae451d33c56b" exitCode=0 Jul 10 21:11:33.760415 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:33.760421 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr" Jul 10 21:11:33.760680 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:33.760467 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr" event={"ID":"179f2c3a-8342-4e7b-b626-1de15c0bc374","Type":"ContainerDied","Data":"af9b0addfe5425eba60bdad5f0de5a779431f44c3ca8a318e05bae451d33c56b"} Jul 10 21:11:33.760680 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:33.760559 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr" event={"ID":"179f2c3a-8342-4e7b-b626-1de15c0bc374","Type":"ContainerDied","Data":"02a3f5ec52060545667d6029669b720798a15d123b850d8fd0e1be5f45b4d2fa"} Jul 10 21:11:33.760680 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:33.760588 2530 scope.go:117] "RemoveContainer" containerID="af9b0addfe5425eba60bdad5f0de5a779431f44c3ca8a318e05bae451d33c56b" Jul 10 21:11:33.770088 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:33.770069 2530 scope.go:117] "RemoveContainer" containerID="af9b0addfe5425eba60bdad5f0de5a779431f44c3ca8a318e05bae451d33c56b" Jul 10 21:11:33.770350 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:11:33.770326 2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af9b0addfe5425eba60bdad5f0de5a779431f44c3ca8a318e05bae451d33c56b\": container with ID starting with af9b0addfe5425eba60bdad5f0de5a779431f44c3ca8a318e05bae451d33c56b not found: ID does not exist" containerID="af9b0addfe5425eba60bdad5f0de5a779431f44c3ca8a318e05bae451d33c56b" Jul 10 21:11:33.770410 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:33.770362 2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af9b0addfe5425eba60bdad5f0de5a779431f44c3ca8a318e05bae451d33c56b"} err="failed to get container status \"af9b0addfe5425eba60bdad5f0de5a779431f44c3ca8a318e05bae451d33c56b\": rpc error: code = NotFound desc = could not find container \"af9b0addfe5425eba60bdad5f0de5a779431f44c3ca8a318e05bae451d33c56b\": container with ID starting with af9b0addfe5425eba60bdad5f0de5a779431f44c3ca8a318e05bae451d33c56b not found: ID does not exist" Jul 10 21:11:33.781639 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:33.781615 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr"] Jul 10 21:11:33.784001 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:33.783982 2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0b11f5-5dcdc95bbc-nlmxr"] Jul 10 21:11:35.774272 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:35.774238 2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="179f2c3a-8342-4e7b-b626-1de15c0bc374" path="/var/lib/kubelet/pods/179f2c3a-8342-4e7b-b626-1de15c0bc374/volumes" Jul 10 21:11:41.909752 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:41.909722 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz"] Jul 10 21:11:41.910168 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:41.910072 2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be794ce1-bf62-447a-ae1e-a4612c5f115d" containerName="maas-api" Jul 10 21:11:41.910168 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:41.910084 2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="be794ce1-bf62-447a-ae1e-a4612c5f115d" containerName="maas-api" Jul 10 21:11:41.910168 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:41.910097 2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="179f2c3a-8342-4e7b-b626-1de15c0bc374" containerName="maas-api" Jul 10 21:11:41.910168 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:41.910103 2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="179f2c3a-8342-4e7b-b626-1de15c0bc374" containerName="maas-api" Jul 10 21:11:41.910168 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:41.910149 2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="179f2c3a-8342-4e7b-b626-1de15c0bc374" containerName="maas-api" Jul 10 21:11:41.910168 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:41.910157 2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="be794ce1-bf62-447a-ae1e-a4612c5f115d" containerName="maas-api" Jul 10 21:11:41.914640 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:41.914614 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz" Jul 10 21:11:41.916876 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:41.916845 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-01037d-serving-cert\"" Jul 10 21:11:41.917003 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:41.916969 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pm7gf\"" Jul 10 21:11:41.924753 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:41.924722 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz"] Jul 10 21:11:42.089377 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:42.089333 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab3b2928-b6f9-42b0-b9bb-9ed8867923ce-maas-api-tls\") pod \"maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz\" (UID: \"ab3b2928-b6f9-42b0-b9bb-9ed8867923ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz" Jul 10 21:11:42.089377 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:42.089381 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdp69\" (UniqueName: \"kubernetes.io/projected/ab3b2928-b6f9-42b0-b9bb-9ed8867923ce-kube-api-access-sdp69\") pod \"maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz\" (UID: \"ab3b2928-b6f9-42b0-b9bb-9ed8867923ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz" Jul 10 21:11:42.089608 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:42.089410 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3b2928-b6f9-42b0-b9bb-9ed8867923ce-openshift-service-ca\") pod \"maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz\" (UID: \"ab3b2928-b6f9-42b0-b9bb-9ed8867923ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz" Jul 10 21:11:42.191017 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:42.190914 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab3b2928-b6f9-42b0-b9bb-9ed8867923ce-maas-api-tls\") pod \"maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz\" (UID: \"ab3b2928-b6f9-42b0-b9bb-9ed8867923ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz" Jul 10 21:11:42.191017 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:42.190970 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdp69\" (UniqueName: \"kubernetes.io/projected/ab3b2928-b6f9-42b0-b9bb-9ed8867923ce-kube-api-access-sdp69\") pod \"maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz\" (UID: \"ab3b2928-b6f9-42b0-b9bb-9ed8867923ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz" Jul 10 21:11:42.191017 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:42.191009 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3b2928-b6f9-42b0-b9bb-9ed8867923ce-openshift-service-ca\") pod \"maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz\" (UID: \"ab3b2928-b6f9-42b0-b9bb-9ed8867923ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz" Jul 10 21:11:42.191669 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:42.191644 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3b2928-b6f9-42b0-b9bb-9ed8867923ce-openshift-service-ca\") pod \"maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz\" (UID: \"ab3b2928-b6f9-42b0-b9bb-9ed8867923ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz" Jul 10 21:11:42.193375 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:42.193347 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab3b2928-b6f9-42b0-b9bb-9ed8867923ce-maas-api-tls\") pod \"maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz\" (UID: \"ab3b2928-b6f9-42b0-b9bb-9ed8867923ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz" Jul 10 21:11:42.199044 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:42.199022 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdp69\" (UniqueName: \"kubernetes.io/projected/ab3b2928-b6f9-42b0-b9bb-9ed8867923ce-kube-api-access-sdp69\") pod \"maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz\" (UID: \"ab3b2928-b6f9-42b0-b9bb-9ed8867923ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz" Jul 10 21:11:42.225912 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:42.225870 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz" Jul 10 21:11:42.359034 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:42.358870 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz"] Jul 10 21:11:42.361597 ip-10-0-139-64 kubenswrapper[2530]: W0710 21:11:42.361572 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3b2928_b6f9_42b0_b9bb_9ed8867923ce.slice/crio-ba7026a1a6ea5779a5616a59e2c325e4a24244717299d4368bd043a1f75d7e14 WatchSource:0}: Error finding container ba7026a1a6ea5779a5616a59e2c325e4a24244717299d4368bd043a1f75d7e14: Status 404 returned error can't find the container with id ba7026a1a6ea5779a5616a59e2c325e4a24244717299d4368bd043a1f75d7e14 Jul 10 21:11:42.801407 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:42.801377 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz" event={"ID":"ab3b2928-b6f9-42b0-b9bb-9ed8867923ce","Type":"ContainerStarted","Data":"ba7026a1a6ea5779a5616a59e2c325e4a24244717299d4368bd043a1f75d7e14"} Jul 10 21:11:43.806259 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:43.806225 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz" event={"ID":"ab3b2928-b6f9-42b0-b9bb-9ed8867923ce","Type":"ContainerStarted","Data":"078ba7c1ba96aad0af493863fee286624f4911dc58982fa5b2bf09216ef874d8"} Jul 10 21:11:43.806629 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:43.806348 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz" Jul 10 21:11:43.823613 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:43.823562 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz" podStartSLOduration=2.382393884 podStartE2EDuration="2.823547626s" totalImagesPullingTime="441.153742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:11:43.822171464 +0000 UTC m=+2216.600752858" watchObservedRunningTime="2026-07-10 21:11:43.823547626 +0000 UTC m=+2216.602129020" Jul 10 21:11:49.816352 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:49.816325 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz" Jul 10 21:11:49.949885 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:49.949857 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz"] Jul 10 21:11:49.960426 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:49.960391 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz" Jul 10 21:11:49.961530 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:49.961502 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz"] Jul 10 21:11:49.962748 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:49.962723 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-e63f46-serving-cert\"" Jul 10 21:11:50.057522 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:50.057482 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9jzj\" (UniqueName: \"kubernetes.io/projected/02734be2-3829-4793-893b-df26bea6b5ce-kube-api-access-l9jzj\") pod \"maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz\" (UID: \"02734be2-3829-4793-893b-df26bea6b5ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz" Jul 10 21:11:50.057686 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:50.057557 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02734be2-3829-4793-893b-df26bea6b5ce-maas-api-tls\") pod \"maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz\" (UID: \"02734be2-3829-4793-893b-df26bea6b5ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz" Jul 10 21:11:50.057686 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:50.057579 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02734be2-3829-4793-893b-df26bea6b5ce-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz\" (UID: \"02734be2-3829-4793-893b-df26bea6b5ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz" Jul 10 21:11:50.157980 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:50.157950 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02734be2-3829-4793-893b-df26bea6b5ce-maas-api-tls\") pod \"maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz\" (UID: \"02734be2-3829-4793-893b-df26bea6b5ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz" Jul 10 21:11:50.158124 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:50.157986 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02734be2-3829-4793-893b-df26bea6b5ce-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz\" (UID: \"02734be2-3829-4793-893b-df26bea6b5ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz" Jul 10 21:11:50.158124 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:11:50.158097 2530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-serving-cert: secret "maas-api-e2e-shared-b-e63f46-serving-cert" not found Jul 10 21:11:50.158202 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:50.158117 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9jzj\" (UniqueName: \"kubernetes.io/projected/02734be2-3829-4793-893b-df26bea6b5ce-kube-api-access-l9jzj\") pod \"maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz\" (UID: \"02734be2-3829-4793-893b-df26bea6b5ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz" Jul 10 21:11:50.158202 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:11:50.158172 2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02734be2-3829-4793-893b-df26bea6b5ce-maas-api-tls podName:02734be2-3829-4793-893b-df26bea6b5ce nodeName:}" failed. No retries permitted until 2026-07-10 21:11:50.658153257 +0000 UTC m=+2223.436734631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/02734be2-3829-4793-893b-df26bea6b5ce-maas-api-tls") pod "maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz" (UID: "02734be2-3829-4793-893b-df26bea6b5ce") : secret "maas-api-e2e-shared-b-e63f46-serving-cert" not found Jul 10 21:11:50.158555 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:50.158538 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02734be2-3829-4793-893b-df26bea6b5ce-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz\" (UID: \"02734be2-3829-4793-893b-df26bea6b5ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz" Jul 10 21:11:50.166166 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:50.166138 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9jzj\" (UniqueName: \"kubernetes.io/projected/02734be2-3829-4793-893b-df26bea6b5ce-kube-api-access-l9jzj\") pod \"maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz\" (UID: \"02734be2-3829-4793-893b-df26bea6b5ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz" Jul 10 21:11:50.661390 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:50.661360 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02734be2-3829-4793-893b-df26bea6b5ce-maas-api-tls\") pod \"maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz\" (UID: \"02734be2-3829-4793-893b-df26bea6b5ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz" Jul 10 21:11:50.663792 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:50.663767 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02734be2-3829-4793-893b-df26bea6b5ce-maas-api-tls\") pod \"maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz\" (UID: \"02734be2-3829-4793-893b-df26bea6b5ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz" Jul 10 21:11:50.871524 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:50.871490 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz" Jul 10 21:11:51.009516 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:51.009491 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz"] Jul 10 21:11:51.011426 ip-10-0-139-64 kubenswrapper[2530]: W0710 21:11:51.011397 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02734be2_3829_4793_893b_df26bea6b5ce.slice/crio-411ce86c3aafa6dea41908064b95cbd0633cca3ea18ec39c3b6f418792c2cce9 WatchSource:0}: Error finding container 411ce86c3aafa6dea41908064b95cbd0633cca3ea18ec39c3b6f418792c2cce9: Status 404 returned error can't find the container with id 411ce86c3aafa6dea41908064b95cbd0633cca3ea18ec39c3b6f418792c2cce9 Jul 10 21:11:51.842121 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:51.842031 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz" event={"ID":"02734be2-3829-4793-893b-df26bea6b5ce","Type":"ContainerStarted","Data":"8c29ee7f70dd3e2aadb170d3e4ede5eacd0c82722065092161f9c4e2157c9997"} Jul 10 21:11:51.842121 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:51.842080 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz" event={"ID":"02734be2-3829-4793-893b-df26bea6b5ce","Type":"ContainerStarted","Data":"411ce86c3aafa6dea41908064b95cbd0633cca3ea18ec39c3b6f418792c2cce9"} Jul 10 21:11:51.842121 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:51.842099 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz" Jul 10 21:11:51.860000 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:51.859948 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz" podStartSLOduration=2.452333629 podStartE2EDuration="2.859902566s" totalImagesPullingTime="407.568937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:11:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:11:51.858993974 +0000 UTC m=+2224.637575386" watchObservedRunningTime="2026-07-10 21:11:51.859902566 +0000 UTC m=+2224.638483964" Jul 10 21:11:57.852989 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:11:57.852963 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz" Jul 10 21:14:47.836068 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:14:47.836041 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n7kg6_a0fc144e-a4ca-4f83-9126-9366a781f0ad/ovn-acl-logging/0.log" Jul 10 21:14:47.841474 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:14:47.841451 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n7kg6_a0fc144e-a4ca-4f83-9126-9366a781f0ad/ovn-acl-logging/0.log" Jul 10 21:15:00.148713 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:15:00.148674 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728635-mdv5g"] Jul 10 21:15:00.152213 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:15:00.152195 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728635-mdv5g" Jul 10 21:15:00.154804 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:15:00.154777 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b692z\"" Jul 10 21:15:00.155144 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:15:00.155129 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e63f46-29728635-fpbmf"] Jul 10 21:15:00.158408 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:15:00.158389 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-01037d-29728635-qc7fq"] Jul 10 21:15:00.158519 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:15:00.158489 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e63f46-29728635-fpbmf" Jul 10 21:15:00.161620 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:15:00.161600 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-01037d-29728635-qc7fq" Jul 10 21:15:00.162257 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:15:00.162239 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728635-mdv5g" Jul 10 21:15:00.171849 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:15:00.170885 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728635-mdv5g"] Jul 10 21:15:00.173256 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:15:00.173202 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e63f46-29728635-fpbmf"] Jul 10 21:15:00.175025 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:15:00.174960 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e63f46-29728635-fpbmf" Jul 10 21:15:00.198496 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:15:00.198345 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-01037d-29728635-qc7fq"] Jul 10 21:15:00.211507 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:15:00.211480 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-01037d-29728635-qc7fq" Jul 10 21:15:00.338141 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:15:00.338105 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728635-mdv5g"] Jul 10 21:15:00.342166 ip-10-0-139-64 kubenswrapper[2530]: W0710 21:15:00.341492 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccddf0a5_97b4_40bc_870a_a760cb7de9dd.slice/crio-ec47e3f8d1a2fb33e9600a3e5d2d23c60de63785427a3a73ffa154d3097e51d2 WatchSource:0}: Error finding container ec47e3f8d1a2fb33e9600a3e5d2d23c60de63785427a3a73ffa154d3097e51d2: Status 404 returned error can't find the container with id ec47e3f8d1a2fb33e9600a3e5d2d23c60de63785427a3a73ffa154d3097e51d2 Jul 10 21:15:00.370526 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:15:00.370497 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e63f46-29728635-fpbmf"] Jul 10 21:15:00.376970 ip-10-0-139-64 kubenswrapper[2530]: W0710 21:15:00.376935 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c8f8cb1_a4ea_4340_9f00_338e8f0e41f1.slice/crio-58e4152fefcf4407ec581037016fabdc47be746c4ca49faaf56fe748869e2a4b WatchSource:0}: Error finding container 58e4152fefcf4407ec581037016fabdc47be746c4ca49faaf56fe748869e2a4b: Status 404 returned error can't find the container with id 58e4152fefcf4407ec581037016fabdc47be746c4ca49faaf56fe748869e2a4b Jul 10 21:15:00.399020 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:15:00.398960 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-01037d-29728635-qc7fq"] Jul 10 21:15:00.404921 ip-10-0-139-64 kubenswrapper[2530]: W0710 21:15:00.404893 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91497d4f_91fc_49b0_b1a1_47d2ddd8c5ff.slice/crio-f695a7b3fef1f1afbcb8061f2978a85e930388451b3d94b5e14e013d5ed53008 WatchSource:0}: Error finding container f695a7b3fef1f1afbcb8061f2978a85e930388451b3d94b5e14e013d5ed53008: Status 404 returned error can't find the container with id f695a7b3fef1f1afbcb8061f2978a85e930388451b3d94b5e14e013d5ed53008 Jul 10 21:15:00.662756 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:15:00.662659 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-01037d-29728635-qc7fq" event={"ID":"91497d4f-91fc-49b0-b1a1-47d2ddd8c5ff","Type":"ContainerStarted","Data":"f695a7b3fef1f1afbcb8061f2978a85e930388451b3d94b5e14e013d5ed53008"} Jul 10 21:15:00.663937 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:15:00.663911 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e63f46-29728635-fpbmf" event={"ID":"4c8f8cb1-a4ea-4340-9f00-338e8f0e41f1","Type":"ContainerStarted","Data":"58e4152fefcf4407ec581037016fabdc47be746c4ca49faaf56fe748869e2a4b"} Jul 10 21:15:00.665305 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:15:00.665282 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728635-mdv5g" event={"ID":"ccddf0a5-97b4-40bc-870a-a760cb7de9dd","Type":"ContainerStarted","Data":"87429a469794e733ef062e116ef1eeacccc769430f63c59cc1b85afbcd3412a0"} Jul 10 21:15:00.665409 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:15:00.665329 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728635-mdv5g" event={"ID":"ccddf0a5-97b4-40bc-870a-a760cb7de9dd","Type":"ContainerStarted","Data":"ec47e3f8d1a2fb33e9600a3e5d2d23c60de63785427a3a73ffa154d3097e51d2"} Jul 10 21:15:00.681770 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:15:00.681717 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728635-mdv5g" podStartSLOduration=0.681700769 podStartE2EDuration="681.700769ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:15:00.680758628 +0000 UTC m=+2413.459340023" watchObservedRunningTime="2026-07-10 21:15:00.681700769 +0000 UTC m=+2413.460282159" Jul 10 21:15:01.671186 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:15:01.671145 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-01037d-29728635-qc7fq" event={"ID":"91497d4f-91fc-49b0-b1a1-47d2ddd8c5ff","Type":"ContainerStarted","Data":"92608fa696a33583e2d7e26c0535620355eade38d6a20e3224244583c7093e0b"} Jul 10 21:15:01.672789 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:15:01.672762 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e63f46-29728635-fpbmf" event={"ID":"4c8f8cb1-a4ea-4340-9f00-338e8f0e41f1","Type":"ContainerStarted","Data":"8d705b67497f7f45fc9ef336e4c6f08cee41134d06eb362f357cf26a6fc4caa3"} Jul 10 21:15:01.687320 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:15:01.687262 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-01037d-29728635-qc7fq" podStartSLOduration=1.687242473 podStartE2EDuration="1.687242473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:15:01.68684616 +0000 UTC m=+2414.465427552" watchObservedRunningTime="2026-07-10 21:15:01.687242473 +0000 UTC m=+2414.465823867" Jul 10 21:15:01.703548 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:15:01.703499 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e63f46-29728635-fpbmf" podStartSLOduration=1.703483069 podStartE2EDuration="1.703483069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:15:01.701999656 +0000 UTC m=+2414.480581050" watchObservedRunningTime="2026-07-10 21:15:01.703483069 +0000 UTC m=+2414.482064534" Jul 10 21:16:00.011514 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:00.011471 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-01037d-29728635-qc7fq"] Jul 10 21:16:00.012297 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:00.011685 2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-01037d-29728635-qc7fq" podUID="91497d4f-91fc-49b0-b1a1-47d2ddd8c5ff" containerName="cleanup" containerID="cri-o://92608fa696a33583e2d7e26c0535620355eade38d6a20e3224244583c7093e0b" gracePeriod=30 Jul 10 21:16:00.013927 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:00.013897 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728635-mdv5g"] Jul 10 21:16:00.014126 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:00.014106 2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728635-mdv5g" podUID="ccddf0a5-97b4-40bc-870a-a760cb7de9dd" containerName="cleanup" containerID="cri-o://87429a469794e733ef062e116ef1eeacccc769430f63c59cc1b85afbcd3412a0" gracePeriod=30 Jul 10 21:16:00.015219 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:00.015196 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e63f46-29728635-fpbmf"] Jul 10 21:16:00.015377 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:00.015343 2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e63f46-29728635-fpbmf" podUID="4c8f8cb1-a4ea-4340-9f00-338e8f0e41f1" containerName="cleanup" containerID="cri-o://8d705b67497f7f45fc9ef336e4c6f08cee41134d06eb362f357cf26a6fc4caa3" gracePeriod=30 Jul 10 21:16:30.055137 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:30.055108 2530 generic.go:356] "Generic (PLEG): container finished" podID="91497d4f-91fc-49b0-b1a1-47d2ddd8c5ff" containerID="92608fa696a33583e2d7e26c0535620355eade38d6a20e3224244583c7093e0b" exitCode=137 Jul 10 21:16:30.055448 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:30.055180 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-01037d-29728635-qc7fq" event={"ID":"91497d4f-91fc-49b0-b1a1-47d2ddd8c5ff","Type":"ContainerDied","Data":"92608fa696a33583e2d7e26c0535620355eade38d6a20e3224244583c7093e0b"} Jul 10 21:16:30.215850 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:30.215802 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-01037d-29728635-qc7fq" Jul 10 21:16:30.242337 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:30.242312 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e63f46-29728635-fpbmf" Jul 10 21:16:30.693292 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:30.693268 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728635-mdv5g" Jul 10 21:16:31.060706 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.060664 2530 generic.go:356] "Generic (PLEG): container finished" podID="ccddf0a5-97b4-40bc-870a-a760cb7de9dd" containerID="87429a469794e733ef062e116ef1eeacccc769430f63c59cc1b85afbcd3412a0" exitCode=137 Jul 10 21:16:31.060706 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.060705 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728635-mdv5g" Jul 10 21:16:31.061224 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.060752 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728635-mdv5g" event={"ID":"ccddf0a5-97b4-40bc-870a-a760cb7de9dd","Type":"ContainerDied","Data":"87429a469794e733ef062e116ef1eeacccc769430f63c59cc1b85afbcd3412a0"} Jul 10 21:16:31.061224 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.060806 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728635-mdv5g" event={"ID":"ccddf0a5-97b4-40bc-870a-a760cb7de9dd","Type":"ContainerDied","Data":"ec47e3f8d1a2fb33e9600a3e5d2d23c60de63785427a3a73ffa154d3097e51d2"} Jul 10 21:16:31.061224 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.060852 2530 scope.go:117] "RemoveContainer" containerID="87429a469794e733ef062e116ef1eeacccc769430f63c59cc1b85afbcd3412a0" Jul 10 21:16:31.062149 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.062133 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-01037d-29728635-qc7fq" Jul 10 21:16:31.062228 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.062131 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-01037d-29728635-qc7fq" event={"ID":"91497d4f-91fc-49b0-b1a1-47d2ddd8c5ff","Type":"ContainerDied","Data":"f695a7b3fef1f1afbcb8061f2978a85e930388451b3d94b5e14e013d5ed53008"} Jul 10 21:16:31.063552 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.063523 2530 generic.go:356] "Generic (PLEG): container finished" podID="4c8f8cb1-a4ea-4340-9f00-338e8f0e41f1" containerID="8d705b67497f7f45fc9ef336e4c6f08cee41134d06eb362f357cf26a6fc4caa3" exitCode=137 Jul 10 21:16:31.063670 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.063563 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e63f46-29728635-fpbmf" Jul 10 21:16:31.063670 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.063573 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e63f46-29728635-fpbmf" event={"ID":"4c8f8cb1-a4ea-4340-9f00-338e8f0e41f1","Type":"ContainerDied","Data":"8d705b67497f7f45fc9ef336e4c6f08cee41134d06eb362f357cf26a6fc4caa3"} Jul 10 21:16:31.063670 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.063624 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e63f46-29728635-fpbmf" event={"ID":"4c8f8cb1-a4ea-4340-9f00-338e8f0e41f1","Type":"ContainerDied","Data":"58e4152fefcf4407ec581037016fabdc47be746c4ca49faaf56fe748869e2a4b"} Jul 10 21:16:31.071514 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.071488 2530 scope.go:117] "RemoveContainer" containerID="87429a469794e733ef062e116ef1eeacccc769430f63c59cc1b85afbcd3412a0" Jul 10 21:16:31.072029 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:16:31.071858 2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87429a469794e733ef062e116ef1eeacccc769430f63c59cc1b85afbcd3412a0\": container with ID starting with 87429a469794e733ef062e116ef1eeacccc769430f63c59cc1b85afbcd3412a0 not found: ID does not exist" containerID="87429a469794e733ef062e116ef1eeacccc769430f63c59cc1b85afbcd3412a0" Jul 10 21:16:31.072029 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.071893 2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87429a469794e733ef062e116ef1eeacccc769430f63c59cc1b85afbcd3412a0"} err="failed to get container status \"87429a469794e733ef062e116ef1eeacccc769430f63c59cc1b85afbcd3412a0\": rpc error: code = NotFound desc = could not find container \"87429a469794e733ef062e116ef1eeacccc769430f63c59cc1b85afbcd3412a0\": container with ID starting with 87429a469794e733ef062e116ef1eeacccc769430f63c59cc1b85afbcd3412a0 not found: ID does not exist" Jul 10 21:16:31.072029 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.071916 2530 scope.go:117] "RemoveContainer" containerID="92608fa696a33583e2d7e26c0535620355eade38d6a20e3224244583c7093e0b" Jul 10 21:16:31.079929 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.079909 2530 scope.go:117] "RemoveContainer" containerID="8d705b67497f7f45fc9ef336e4c6f08cee41134d06eb362f357cf26a6fc4caa3" Jul 10 21:16:31.085461 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.085439 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-01037d-29728635-qc7fq"] Jul 10 21:16:31.087757 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.087740 2530 scope.go:117] "RemoveContainer" containerID="8d705b67497f7f45fc9ef336e4c6f08cee41134d06eb362f357cf26a6fc4caa3" Jul 10 21:16:31.088056 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:16:31.088040 2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d705b67497f7f45fc9ef336e4c6f08cee41134d06eb362f357cf26a6fc4caa3\": container with ID starting with 8d705b67497f7f45fc9ef336e4c6f08cee41134d06eb362f357cf26a6fc4caa3 not found: ID does not exist" containerID="8d705b67497f7f45fc9ef336e4c6f08cee41134d06eb362f357cf26a6fc4caa3" Jul 10 21:16:31.088113 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.088063 2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d705b67497f7f45fc9ef336e4c6f08cee41134d06eb362f357cf26a6fc4caa3"} err="failed to get container status \"8d705b67497f7f45fc9ef336e4c6f08cee41134d06eb362f357cf26a6fc4caa3\": rpc error: code = NotFound desc = could not find container \"8d705b67497f7f45fc9ef336e4c6f08cee41134d06eb362f357cf26a6fc4caa3\": container with ID starting with 8d705b67497f7f45fc9ef336e4c6f08cee41134d06eb362f357cf26a6fc4caa3 not found: ID does not exist" Jul 10 21:16:31.090992 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.090970 2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-01037d-29728635-qc7fq"] Jul 10 21:16:31.101522 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.101501 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728635-mdv5g"] Jul 10 21:16:31.105274 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.105255 2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728635-mdv5g"] Jul 10 21:16:31.115914 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.115880 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e63f46-29728635-fpbmf"] Jul 10 21:16:31.120343 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.120322 2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-e63f46-29728635-fpbmf"] Jul 10 21:16:31.772590 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.772556 2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c8f8cb1-a4ea-4340-9f00-338e8f0e41f1" path="/var/lib/kubelet/pods/4c8f8cb1-a4ea-4340-9f00-338e8f0e41f1/volumes" Jul 10 21:16:31.772815 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.772801 2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91497d4f-91fc-49b0-b1a1-47d2ddd8c5ff" path="/var/lib/kubelet/pods/91497d4f-91fc-49b0-b1a1-47d2ddd8c5ff/volumes" Jul 10 21:16:31.773045 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:16:31.773033 2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccddf0a5-97b4-40bc-870a-a760cb7de9dd" path="/var/lib/kubelet/pods/ccddf0a5-97b4-40bc-870a-a760cb7de9dd/volumes" Jul 10 21:17:48.480552 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:48.480516 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz"] Jul 10 21:17:48.481451 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:48.481404 2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz" podUID="ab3b2928-b6f9-42b0-b9bb-9ed8867923ce" containerName="maas-api" containerID="cri-o://078ba7c1ba96aad0af493863fee286624f4911dc58982fa5b2bf09216ef874d8" gracePeriod=30 Jul 10 21:17:48.933077 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:48.933051 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz" Jul 10 21:17:49.090704 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:49.090604 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdp69\" (UniqueName: \"kubernetes.io/projected/ab3b2928-b6f9-42b0-b9bb-9ed8867923ce-kube-api-access-sdp69\") pod \"ab3b2928-b6f9-42b0-b9bb-9ed8867923ce\" (UID: \"ab3b2928-b6f9-42b0-b9bb-9ed8867923ce\") " Jul 10 21:17:49.090704 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:49.090658 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3b2928-b6f9-42b0-b9bb-9ed8867923ce-openshift-service-ca\") pod \"ab3b2928-b6f9-42b0-b9bb-9ed8867923ce\" (UID: \"ab3b2928-b6f9-42b0-b9bb-9ed8867923ce\") " Jul 10 21:17:49.090977 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:49.090735 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab3b2928-b6f9-42b0-b9bb-9ed8867923ce-maas-api-tls\") pod \"ab3b2928-b6f9-42b0-b9bb-9ed8867923ce\" (UID: \"ab3b2928-b6f9-42b0-b9bb-9ed8867923ce\") " Jul 10 21:17:49.091137 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:49.091113 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab3b2928-b6f9-42b0-b9bb-9ed8867923ce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ab3b2928-b6f9-42b0-b9bb-9ed8867923ce" (UID: "ab3b2928-b6f9-42b0-b9bb-9ed8867923ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:17:49.092770 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:49.092742 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab3b2928-b6f9-42b0-b9bb-9ed8867923ce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ab3b2928-b6f9-42b0-b9bb-9ed8867923ce" (UID: "ab3b2928-b6f9-42b0-b9bb-9ed8867923ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:17:49.092888 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:49.092788 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab3b2928-b6f9-42b0-b9bb-9ed8867923ce-kube-api-access-sdp69" (OuterVolumeSpecName: "kube-api-access-sdp69") pod "ab3b2928-b6f9-42b0-b9bb-9ed8867923ce" (UID: "ab3b2928-b6f9-42b0-b9bb-9ed8867923ce"). InnerVolumeSpecName "kube-api-access-sdp69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:17:49.191374 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:49.191336 2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab3b2928-b6f9-42b0-b9bb-9ed8867923ce-maas-api-tls\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:17:49.191374 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:49.191369 2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdp69\" (UniqueName: \"kubernetes.io/projected/ab3b2928-b6f9-42b0-b9bb-9ed8867923ce-kube-api-access-sdp69\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:17:49.191374 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:49.191380 2530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3b2928-b6f9-42b0-b9bb-9ed8867923ce-openshift-service-ca\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:17:49.402545 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:49.402516 2530 generic.go:356] "Generic (PLEG): container finished" podID="ab3b2928-b6f9-42b0-b9bb-9ed8867923ce" containerID="078ba7c1ba96aad0af493863fee286624f4911dc58982fa5b2bf09216ef874d8" exitCode=0 Jul 10 21:17:49.402712 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:49.402557 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz" Jul 10 21:17:49.402712 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:49.402563 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz" event={"ID":"ab3b2928-b6f9-42b0-b9bb-9ed8867923ce","Type":"ContainerDied","Data":"078ba7c1ba96aad0af493863fee286624f4911dc58982fa5b2bf09216ef874d8"} Jul 10 21:17:49.402712 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:49.402605 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz" event={"ID":"ab3b2928-b6f9-42b0-b9bb-9ed8867923ce","Type":"ContainerDied","Data":"ba7026a1a6ea5779a5616a59e2c325e4a24244717299d4368bd043a1f75d7e14"} Jul 10 21:17:49.402712 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:49.402627 2530 scope.go:117] "RemoveContainer" containerID="078ba7c1ba96aad0af493863fee286624f4911dc58982fa5b2bf09216ef874d8" Jul 10 21:17:49.411124 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:49.411108 2530 scope.go:117] "RemoveContainer" containerID="078ba7c1ba96aad0af493863fee286624f4911dc58982fa5b2bf09216ef874d8" Jul 10 21:17:49.411361 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:17:49.411344 2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"078ba7c1ba96aad0af493863fee286624f4911dc58982fa5b2bf09216ef874d8\": container with ID starting with 078ba7c1ba96aad0af493863fee286624f4911dc58982fa5b2bf09216ef874d8 not found: ID does not exist" containerID="078ba7c1ba96aad0af493863fee286624f4911dc58982fa5b2bf09216ef874d8" Jul 10 21:17:49.411409 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:49.411371 2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"078ba7c1ba96aad0af493863fee286624f4911dc58982fa5b2bf09216ef874d8"} err="failed to get container status \"078ba7c1ba96aad0af493863fee286624f4911dc58982fa5b2bf09216ef874d8\": rpc error: code = NotFound desc = could not find container \"078ba7c1ba96aad0af493863fee286624f4911dc58982fa5b2bf09216ef874d8\": container with ID starting with 078ba7c1ba96aad0af493863fee286624f4911dc58982fa5b2bf09216ef874d8 not found: ID does not exist" Jul 10 21:17:49.423274 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:49.423246 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz"] Jul 10 21:17:49.428409 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:49.428385 2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-01037d-7fcc886b47-xm7pz"] Jul 10 21:17:49.772530 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:49.772448 2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab3b2928-b6f9-42b0-b9bb-9ed8867923ce" path="/var/lib/kubelet/pods/ab3b2928-b6f9-42b0-b9bb-9ed8867923ce/volumes" Jul 10 21:17:55.654199 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:55.654133 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz"] Jul 10 21:17:55.654651 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:55.654425 2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz" podUID="02734be2-3829-4793-893b-df26bea6b5ce" containerName="maas-api" containerID="cri-o://8c29ee7f70dd3e2aadb170d3e4ede5eacd0c82722065092161f9c4e2157c9997" gracePeriod=30 Jul 10 21:17:55.900308 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:55.900277 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz" Jul 10 21:17:56.052908 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:56.052801 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02734be2-3829-4793-893b-df26bea6b5ce-openshift-service-ca\") pod \"02734be2-3829-4793-893b-df26bea6b5ce\" (UID: \"02734be2-3829-4793-893b-df26bea6b5ce\") " Jul 10 21:17:56.052908 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:56.052897 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9jzj\" (UniqueName: \"kubernetes.io/projected/02734be2-3829-4793-893b-df26bea6b5ce-kube-api-access-l9jzj\") pod \"02734be2-3829-4793-893b-df26bea6b5ce\" (UID: \"02734be2-3829-4793-893b-df26bea6b5ce\") " Jul 10 21:17:56.053114 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:56.052953 2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02734be2-3829-4793-893b-df26bea6b5ce-maas-api-tls\") pod \"02734be2-3829-4793-893b-df26bea6b5ce\" (UID: \"02734be2-3829-4793-893b-df26bea6b5ce\") " Jul 10 21:17:56.053234 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:56.053206 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02734be2-3829-4793-893b-df26bea6b5ce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "02734be2-3829-4793-893b-df26bea6b5ce" (UID: "02734be2-3829-4793-893b-df26bea6b5ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 21:17:56.054952 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:56.054920 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02734be2-3829-4793-893b-df26bea6b5ce-kube-api-access-l9jzj" (OuterVolumeSpecName: "kube-api-access-l9jzj") pod "02734be2-3829-4793-893b-df26bea6b5ce" (UID: "02734be2-3829-4793-893b-df26bea6b5ce"). InnerVolumeSpecName "kube-api-access-l9jzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 21:17:56.055055 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:56.055026 2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02734be2-3829-4793-893b-df26bea6b5ce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "02734be2-3829-4793-893b-df26bea6b5ce" (UID: "02734be2-3829-4793-893b-df26bea6b5ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 21:17:56.153700 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:56.153664 2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02734be2-3829-4793-893b-df26bea6b5ce-maas-api-tls\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:17:56.153700 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:56.153694 2530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02734be2-3829-4793-893b-df26bea6b5ce-openshift-service-ca\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:17:56.153700 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:56.153704 2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9jzj\" (UniqueName: \"kubernetes.io/projected/02734be2-3829-4793-893b-df26bea6b5ce-kube-api-access-l9jzj\") on node \"ip-10-0-139-64.ec2.internal\" DevicePath \"\"" Jul 10 21:17:56.433214 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:56.433175 2530 generic.go:356] "Generic (PLEG): container finished" podID="02734be2-3829-4793-893b-df26bea6b5ce" containerID="8c29ee7f70dd3e2aadb170d3e4ede5eacd0c82722065092161f9c4e2157c9997" exitCode=0 Jul 10 21:17:56.433384 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:56.433221 2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz" Jul 10 21:17:56.433384 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:56.433251 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz" event={"ID":"02734be2-3829-4793-893b-df26bea6b5ce","Type":"ContainerDied","Data":"8c29ee7f70dd3e2aadb170d3e4ede5eacd0c82722065092161f9c4e2157c9997"} Jul 10 21:17:56.433384 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:56.433309 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz" event={"ID":"02734be2-3829-4793-893b-df26bea6b5ce","Type":"ContainerDied","Data":"411ce86c3aafa6dea41908064b95cbd0633cca3ea18ec39c3b6f418792c2cce9"} Jul 10 21:17:56.433384 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:56.433332 2530 scope.go:117] "RemoveContainer" containerID="8c29ee7f70dd3e2aadb170d3e4ede5eacd0c82722065092161f9c4e2157c9997" Jul 10 21:17:56.442089 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:56.442069 2530 scope.go:117] "RemoveContainer" containerID="8c29ee7f70dd3e2aadb170d3e4ede5eacd0c82722065092161f9c4e2157c9997" Jul 10 21:17:56.442369 ip-10-0-139-64 kubenswrapper[2530]: E0710 21:17:56.442349 2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c29ee7f70dd3e2aadb170d3e4ede5eacd0c82722065092161f9c4e2157c9997\": container with ID starting with 8c29ee7f70dd3e2aadb170d3e4ede5eacd0c82722065092161f9c4e2157c9997 not found: ID does not exist" containerID="8c29ee7f70dd3e2aadb170d3e4ede5eacd0c82722065092161f9c4e2157c9997" Jul 10 21:17:56.442414 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:56.442380 2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c29ee7f70dd3e2aadb170d3e4ede5eacd0c82722065092161f9c4e2157c9997"} err="failed to get container status \"8c29ee7f70dd3e2aadb170d3e4ede5eacd0c82722065092161f9c4e2157c9997\": rpc error: code = NotFound desc = could not find container \"8c29ee7f70dd3e2aadb170d3e4ede5eacd0c82722065092161f9c4e2157c9997\": container with ID starting with 8c29ee7f70dd3e2aadb170d3e4ede5eacd0c82722065092161f9c4e2157c9997 not found: ID does not exist" Jul 10 21:17:56.455657 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:56.455628 2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz"] Jul 10 21:17:56.457012 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:56.456992 2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e63f46-56d6fd7857-j8hcz"] Jul 10 21:17:57.772243 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:17:57.772200 2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02734be2-3829-4793-893b-df26bea6b5ce" path="/var/lib/kubelet/pods/02734be2-3829-4793-893b-df26bea6b5ce/volumes" Jul 10 21:18:07.103366 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:07.103331 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6bc6856dbb-pr9zj_fbc0bae2-e7f8-446d-b6a5-f99bdc987030/manager/0.log" Jul 10 21:18:07.441346 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:07.441322 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d5798c7df-9t9rk_ed408813-2dcb-4d08-bec9-879a3018d2b8/copy-manifests/0.log" Jul 10 21:18:07.447790 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:07.447768 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d5798c7df-9t9rk_ed408813-2dcb-4d08-bec9-879a3018d2b8/manager/0.log" Jul 10 21:18:08.044964 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:08.044936 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-d59fd97f4-5k7v2_63d8f1cc-7ae8-49ad-861f-ddeb0f64f4b6/maas-api/0.log" Jul 10 21:18:08.155653 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:08.155623 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-rjzz6_5afc7bda-a2d6-4dc4-a33d-10792e5a347a/postgres/0.log" Jul 10 21:18:10.711258 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:10.711180 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-9hnkt_b6361b68-16e0-4d13-8439-095977bfcedc/discovery/0.log" Jul 10 21:18:11.173509 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:11.173479 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-dxh94_a7ddedd9-ceb8-4c42-9e9c-d760fed67935/payload-pre-processing/0.log" Jul 10 21:18:11.294768 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:11.294733 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-ns2gt_5bccfbea-5309-4201-8241-2a4236b00cf8/payload-processing/0.log" Jul 10 21:18:11.871579 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:11.871551 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-cfbjd_23d1b156-b04a-417e-be23-0d58b14cf223/main/0.log" Jul 10 21:18:11.982375 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:11.982326 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-sv75x_e23d257b-40b9-469f-868d-0715c3006b72/main/0.log" Jul 10 21:18:18.871001 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:18.870964 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lmvbr_7da12ef6-d8c0-4c2c-823a-33c8d885e222/global-pull-secret-syncer/0.log" Jul 10 21:18:19.006062 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:19.006030 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gx5mn_e4d35ba8-662a-461f-b95a-7e3c54034a1f/konnectivity-agent/0.log" Jul 10 21:18:19.111007 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:19.110981 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-64.ec2.internal_9e4bdc1f22e9646521fc55e80037564b/haproxy/0.log" Jul 10 21:18:26.632598 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:26.632572 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9gqvk_f8317950-0804-43bf-82e3-e374099dc27b/node-exporter/0.log" Jul 10 21:18:26.680622 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:26.680588 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9gqvk_f8317950-0804-43bf-82e3-e374099dc27b/kube-rbac-proxy/0.log" Jul 10 21:18:26.742364 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:26.742335 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9gqvk_f8317950-0804-43bf-82e3-e374099dc27b/init-textfile/0.log" Jul 10 21:18:27.083561 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:27.083533 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_207e13a4-5747-41b8-a458-025de50e8028/prometheus/0.log" Jul 10 21:18:27.114177 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:27.114146 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_207e13a4-5747-41b8-a458-025de50e8028/config-reloader/0.log" Jul 10 21:18:27.135675 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:27.135593 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_207e13a4-5747-41b8-a458-025de50e8028/thanos-sidecar/0.log" Jul 10 21:18:27.158757 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:27.158727 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_207e13a4-5747-41b8-a458-025de50e8028/kube-rbac-proxy-web/0.log" Jul 10 21:18:27.180049 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:27.180023 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_207e13a4-5747-41b8-a458-025de50e8028/kube-rbac-proxy/0.log" Jul 10 21:18:27.206741 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:27.206704 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_207e13a4-5747-41b8-a458-025de50e8028/kube-rbac-proxy-thanos/0.log" Jul 10 21:18:27.228490 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:27.228468 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_207e13a4-5747-41b8-a458-025de50e8028/init-config-reloader/0.log" Jul 10 21:18:27.313649 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:27.313618 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-zzmxl_f279cd39-20ee-46df-8f56-13e2b6fe82e8/prometheus-operator-admission-webhook/0.log" Jul 10 21:18:28.815006 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.814972 2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6snqv/perf-node-gather-daemonset-8gs6d"] Jul 10 21:18:28.815408 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.815277 2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91497d4f-91fc-49b0-b1a1-47d2ddd8c5ff" containerName="cleanup" Jul 10 21:18:28.815408 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.815288 2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="91497d4f-91fc-49b0-b1a1-47d2ddd8c5ff" containerName="cleanup" Jul 10 21:18:28.815408 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.815298 2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c8f8cb1-a4ea-4340-9f00-338e8f0e41f1" containerName="cleanup" Jul 10 21:18:28.815408 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.815303 2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c8f8cb1-a4ea-4340-9f00-338e8f0e41f1" containerName="cleanup" Jul 10 21:18:28.815408 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.815319 2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccddf0a5-97b4-40bc-870a-a760cb7de9dd" containerName="cleanup" Jul 10 21:18:28.815408 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.815328 2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccddf0a5-97b4-40bc-870a-a760cb7de9dd" containerName="cleanup" Jul 10 21:18:28.815408 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.815336 2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab3b2928-b6f9-42b0-b9bb-9ed8867923ce" containerName="maas-api" Jul 10 21:18:28.815408 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.815340 2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab3b2928-b6f9-42b0-b9bb-9ed8867923ce" containerName="maas-api" Jul 10 21:18:28.815408 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.815347 2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02734be2-3829-4793-893b-df26bea6b5ce" containerName="maas-api" Jul 10 21:18:28.815408 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.815352 2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="02734be2-3829-4793-893b-df26bea6b5ce" containerName="maas-api" Jul 10 21:18:28.815743 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.815424 2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab3b2928-b6f9-42b0-b9bb-9ed8867923ce" containerName="maas-api" Jul 10 21:18:28.815743 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.815432 2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="02734be2-3829-4793-893b-df26bea6b5ce" containerName="maas-api" Jul 10 21:18:28.815743 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.815440 2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="91497d4f-91fc-49b0-b1a1-47d2ddd8c5ff" containerName="cleanup" Jul 10 21:18:28.815743 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.815446 2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccddf0a5-97b4-40bc-870a-a760cb7de9dd" containerName="cleanup" Jul 10 21:18:28.815743 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.815451 2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c8f8cb1-a4ea-4340-9f00-338e8f0e41f1" containerName="cleanup" Jul 10 21:18:28.818024 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.818004 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6snqv/perf-node-gather-daemonset-8gs6d" Jul 10 21:18:28.820347 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.820328 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6snqv\"/\"kube-root-ca.crt\"" Jul 10 21:18:28.821030 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.821012 2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6snqv\"/\"default-dockercfg-5mrqk\"" Jul 10 21:18:28.821115 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.821021 2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6snqv\"/\"openshift-service-ca.crt\"" Jul 10 21:18:28.826063 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.826043 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6snqv/perf-node-gather-daemonset-8gs6d"] Jul 10 21:18:28.931220 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.931186 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9-proc\") pod \"perf-node-gather-daemonset-8gs6d\" (UID: \"9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9\") " pod="openshift-must-gather-6snqv/perf-node-gather-daemonset-8gs6d" Jul 10 21:18:28.931220 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.931217 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9-podres\") pod \"perf-node-gather-daemonset-8gs6d\" (UID: \"9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9\") " pod="openshift-must-gather-6snqv/perf-node-gather-daemonset-8gs6d" Jul 10 21:18:28.931442 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.931236 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7np4\" (UniqueName: \"kubernetes.io/projected/9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9-kube-api-access-z7np4\") pod \"perf-node-gather-daemonset-8gs6d\" (UID: \"9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9\") " pod="openshift-must-gather-6snqv/perf-node-gather-daemonset-8gs6d" Jul 10 21:18:28.931442 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.931352 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9-sys\") pod \"perf-node-gather-daemonset-8gs6d\" (UID: \"9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9\") " pod="openshift-must-gather-6snqv/perf-node-gather-daemonset-8gs6d" Jul 10 21:18:28.931442 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:28.931386 2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9-lib-modules\") pod \"perf-node-gather-daemonset-8gs6d\" (UID: \"9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9\") " pod="openshift-must-gather-6snqv/perf-node-gather-daemonset-8gs6d" Jul 10 21:18:29.031958 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:29.031913 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9-sys\") pod \"perf-node-gather-daemonset-8gs6d\" (UID: \"9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9\") " pod="openshift-must-gather-6snqv/perf-node-gather-daemonset-8gs6d" Jul 10 21:18:29.031958 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:29.031966 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9-lib-modules\") pod \"perf-node-gather-daemonset-8gs6d\" (UID: \"9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9\") " pod="openshift-must-gather-6snqv/perf-node-gather-daemonset-8gs6d" Jul 10 21:18:29.032115 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:29.032010 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9-proc\") pod \"perf-node-gather-daemonset-8gs6d\" (UID: \"9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9\") " pod="openshift-must-gather-6snqv/perf-node-gather-daemonset-8gs6d" Jul 10 21:18:29.032115 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:29.032027 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9-podres\") pod \"perf-node-gather-daemonset-8gs6d\" (UID: \"9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9\") " pod="openshift-must-gather-6snqv/perf-node-gather-daemonset-8gs6d" Jul 10 21:18:29.032115 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:29.032043 2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7np4\" (UniqueName: \"kubernetes.io/projected/9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9-kube-api-access-z7np4\") pod \"perf-node-gather-daemonset-8gs6d\" (UID: \"9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9\") " pod="openshift-must-gather-6snqv/perf-node-gather-daemonset-8gs6d" Jul 10 21:18:29.032115 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:29.032105 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9-proc\") pod \"perf-node-gather-daemonset-8gs6d\" (UID: \"9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9\") " pod="openshift-must-gather-6snqv/perf-node-gather-daemonset-8gs6d" Jul 10 21:18:29.032307 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:29.032041 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9-sys\") pod \"perf-node-gather-daemonset-8gs6d\" (UID: \"9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9\") " pod="openshift-must-gather-6snqv/perf-node-gather-daemonset-8gs6d" Jul 10 21:18:29.032307 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:29.032112 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9-podres\") pod \"perf-node-gather-daemonset-8gs6d\" (UID: \"9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9\") " pod="openshift-must-gather-6snqv/perf-node-gather-daemonset-8gs6d" Jul 10 21:18:29.032307 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:29.032164 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9-lib-modules\") pod \"perf-node-gather-daemonset-8gs6d\" (UID: \"9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9\") " pod="openshift-must-gather-6snqv/perf-node-gather-daemonset-8gs6d" Jul 10 21:18:29.040073 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:29.040052 2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7np4\" (UniqueName: \"kubernetes.io/projected/9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9-kube-api-access-z7np4\") pod \"perf-node-gather-daemonset-8gs6d\" (UID: \"9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9\") " pod="openshift-must-gather-6snqv/perf-node-gather-daemonset-8gs6d" Jul 10 21:18:29.128308 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:29.128277 2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6snqv/perf-node-gather-daemonset-8gs6d" Jul 10 21:18:29.254589 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:29.254567 2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6snqv/perf-node-gather-daemonset-8gs6d"] Jul 10 21:18:29.256806 ip-10-0-139-64 kubenswrapper[2530]: W0710 21:18:29.256777 2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9ad1ee06_fe6f_4fb7_bde8_8f1bbce485c9.slice/crio-5e278ec270f3d4a8cea69698f826d46c8a29933954d1ea2467fcb894362bedf6 WatchSource:0}: Error finding container 5e278ec270f3d4a8cea69698f826d46c8a29933954d1ea2467fcb894362bedf6: Status 404 returned error can't find the container with id 5e278ec270f3d4a8cea69698f826d46c8a29933954d1ea2467fcb894362bedf6 Jul 10 21:18:29.573151 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:29.573053 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6snqv/perf-node-gather-daemonset-8gs6d" event={"ID":"9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9","Type":"ContainerStarted","Data":"cbfd37e5cd4ebf8e5ac4ec888f7035aada576f43094dc55fa2e1e716eec8494e"} Jul 10 21:18:29.573151 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:29.573105 2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6snqv/perf-node-gather-daemonset-8gs6d" event={"ID":"9ad1ee06-fe6f-4fb7-bde8-8f1bbce485c9","Type":"ContainerStarted","Data":"5e278ec270f3d4a8cea69698f826d46c8a29933954d1ea2467fcb894362bedf6"} Jul 10 21:18:29.573151 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:29.573139 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6snqv/perf-node-gather-daemonset-8gs6d" Jul 10 21:18:29.595032 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:29.594986 2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6snqv/perf-node-gather-daemonset-8gs6d" podStartSLOduration=1.594971062 podStartE2EDuration="1.594971062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 21:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:18:29.593248953 +0000 UTC m=+2622.371830450" watchObservedRunningTime="2026-07-10 21:18:29.594971062 +0000 UTC m=+2622.373552455" Jul 10 21:18:31.418350 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:31.418307 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2fbhq_2974e2ef-e853-4b53-8684-35405b1110fe/dns/0.log" Jul 10 21:18:31.440296 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:31.440274 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2fbhq_2974e2ef-e853-4b53-8684-35405b1110fe/kube-rbac-proxy/0.log" Jul 10 21:18:31.594040 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:31.594009 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kpwbz_12401ef9-0e51-4dc0-8e83-ccb7fe160a6e/dns-node-resolver/0.log" Jul 10 21:18:32.185422 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:32.185377 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w9q99_7b2743f9-3a33-4dfa-a239-dbd0c983bd20/node-ca/0.log" Jul 10 21:18:33.350484 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:33.350453 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-9hnkt_b6361b68-16e0-4d13-8439-095977bfcedc/discovery/0.log" Jul 10 21:18:33.502561 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:33.502532 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-dxh94_a7ddedd9-ceb8-4c42-9e9c-d760fed67935/payload-pre-processing/0.log" Jul 10 21:18:33.548087 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:33.548061 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-ns2gt_5bccfbea-5309-4201-8241-2a4236b00cf8/payload-processing/0.log" Jul 10 21:18:34.305162 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:34.305134 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lz84m_cf415b33-0aa4-49c9-a1bb-9f63108af200/serve-healthcheck-canary/0.log" Jul 10 21:18:34.841325 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:34.841297 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jspvl_eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5/kube-rbac-proxy/0.log" Jul 10 21:18:34.864664 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:34.864640 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jspvl_eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5/exporter/0.log" Jul 10 21:18:34.890043 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:34.890020 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jspvl_eb3d7e6c-15fa-4c26-b7c2-7282aa5e71e5/extractor/0.log" Jul 10 21:18:35.586963 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:35.586933 2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6snqv/perf-node-gather-daemonset-8gs6d" Jul 10 21:18:37.471639 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:37.471605 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6bc6856dbb-pr9zj_fbc0bae2-e7f8-446d-b6a5-f99bdc987030/manager/0.log" Jul 10 21:18:37.590137 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:37.590101 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d5798c7df-9t9rk_ed408813-2dcb-4d08-bec9-879a3018d2b8/manager/0.log" Jul 10 21:18:37.610803 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:37.610779 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d5798c7df-9t9rk_ed408813-2dcb-4d08-bec9-879a3018d2b8/copy-manifests/0.log" Jul 10 21:18:39.145399 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:39.145371 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7c5b7cd584-mjlnp_edf04234-9822-4e34-b7fb-905f0598fe27/manager/0.log" Jul 10 21:18:39.644534 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:39.644506 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-9hnkt_b6361b68-16e0-4d13-8439-095977bfcedc/discovery/0.log" Jul 10 21:18:39.756242 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:39.756209 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-dxh94_a7ddedd9-ceb8-4c42-9e9c-d760fed67935/payload-pre-processing/0.log" Jul 10 21:18:39.794080 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:39.794047 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-ns2gt_5bccfbea-5309-4201-8241-2a4236b00cf8/payload-processing/0.log" Jul 10 21:18:41.257256 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:41.257216 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-9hnkt_b6361b68-16e0-4d13-8439-095977bfcedc/discovery/0.log" Jul 10 21:18:41.374244 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:41.374214 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-dxh94_a7ddedd9-ceb8-4c42-9e9c-d760fed67935/payload-pre-processing/0.log" Jul 10 21:18:41.411032 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:41.411000 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-ns2gt_5bccfbea-5309-4201-8241-2a4236b00cf8/payload-processing/0.log" Jul 10 21:18:45.492706 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:45.492674 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-47wft_d6c0b1df-8049-472c-b375-1812f1ff620b/kube-multus-additional-cni-plugins/0.log" Jul 10 21:18:45.514609 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:45.514583 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-47wft_d6c0b1df-8049-472c-b375-1812f1ff620b/egress-router-binary-copy/0.log" Jul 10 21:18:45.537495 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:45.537472 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-47wft_d6c0b1df-8049-472c-b375-1812f1ff620b/cni-plugins/0.log" Jul 10 21:18:45.559139 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:45.559119 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-47wft_d6c0b1df-8049-472c-b375-1812f1ff620b/bond-cni-plugin/0.log" Jul 10 21:18:45.583763 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:45.583742 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-47wft_d6c0b1df-8049-472c-b375-1812f1ff620b/routeoverride-cni/0.log" Jul 10 21:18:45.604341 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:45.604320 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-47wft_d6c0b1df-8049-472c-b375-1812f1ff620b/whereabouts-cni-bincopy/0.log" Jul 10 21:18:45.626494 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:45.626470 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-47wft_d6c0b1df-8049-472c-b375-1812f1ff620b/whereabouts-cni/0.log" Jul 10 21:18:46.012733 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:46.012706 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qmjv5_9b7a55bf-4524-49b1-9147-2e0bdb51d2db/kube-multus/0.log" Jul 10 21:18:46.115287 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:46.115255 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dwvvg_93c55127-6ddc-4b15-844e-8b9585cc5d1e/network-metrics-daemon/0.log" Jul 10 21:18:46.135874 ip-10-0-139-64 kubenswrapper[2530]: I0710 21:18:46.135847 2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dwvvg_93c55127-6ddc-4b15-844e-8b9585cc5d1e/kube-rbac-proxy/0.log"