Sep 10 22:04:58.325211 ip-10-0-137-148 systemd[1]: Starting Kubernetes Kubelet... Sep 10 22:04:58.741187 ip-10-0-137-148 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 22:04:58.741187 ip-10-0-137-148 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 22:04:58.741187 ip-10-0-137-148 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 22:04:58.741187 ip-10-0-137-148 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 22:04:58.741187 ip-10-0-137-148 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 22:04:58.741826 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.741198 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 22:04:58.747074 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747050 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 22:04:58.747074 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747067 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 22:04:58.747074 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747071 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 22:04:58.747074 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747075 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 22:04:58.747074 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747078 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 22:04:58.747074 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747081 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 22:04:58.747292 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747084 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 22:04:58.747292 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747086 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 22:04:58.747292 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747089 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 22:04:58.747292 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747092 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 22:04:58.747292 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747094 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 22:04:58.747292 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747098 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 22:04:58.747292 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747100 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 22:04:58.747292 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747103 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 22:04:58.747292 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747105 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 22:04:58.747292 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747108 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 22:04:58.747292 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747111 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 22:04:58.747292 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747113 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 22:04:58.747292 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747116 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 22:04:58.747292 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747118 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 22:04:58.747292 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747121 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 22:04:58.747292 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747123 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 22:04:58.747292 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747126 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 22:04:58.747292 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747128 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 22:04:58.747292 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747131 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 22:04:58.747292 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747133 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 22:04:58.747762 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747136 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 22:04:58.747762 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747139 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 22:04:58.747762 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747141 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 22:04:58.747762 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747143 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 22:04:58.747762 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747147 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 22:04:58.747762 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747149 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 22:04:58.747762 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747152 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 22:04:58.747762 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747154 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 22:04:58.747762 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747157 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 22:04:58.747762 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747160 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 22:04:58.747762 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747162 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 22:04:58.747762 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747165 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 22:04:58.747762 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747168 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 22:04:58.747762 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747171 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 22:04:58.747762 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747173 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 22:04:58.747762 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747176 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 22:04:58.747762 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747179 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 22:04:58.747762 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747181 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 22:04:58.747762 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747185 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 22:04:58.747762 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747188 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 22:04:58.748264 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747190 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 22:04:58.748264 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747193 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 22:04:58.748264 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747195 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 22:04:58.748264 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747198 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 22:04:58.748264 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747201 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 22:04:58.748264 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747203 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 22:04:58.748264 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747206 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 22:04:58.748264 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747209 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 22:04:58.748264 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747211 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 22:04:58.748264 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747215 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 22:04:58.748264 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747218 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 22:04:58.748264 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747220 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 22:04:58.748264 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747223 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 22:04:58.748264 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747225 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 22:04:58.748264 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747227 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 22:04:58.748264 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747230 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 22:04:58.748264 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747232 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 22:04:58.748264 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747235 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 22:04:58.748264 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747237 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 22:04:58.748710 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747240 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 22:04:58.748710 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747242 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 22:04:58.748710 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747245 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 10 22:04:58.748710 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747247 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 22:04:58.748710 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747249 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 22:04:58.748710 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747252 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 22:04:58.748710 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747255 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 22:04:58.748710 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747260 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 22:04:58.748710 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747264 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 22:04:58.748710 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747267 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 22:04:58.748710 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747270 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 22:04:58.748710 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747272 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 22:04:58.748710 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747275 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 22:04:58.748710 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747278 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 22:04:58.748710 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747282 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 22:04:58.748710 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747285 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 22:04:58.748710 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747289 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 22:04:58.748710 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747291 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 22:04:58.748710 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747294 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 22:04:58.749206 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747297 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 22:04:58.749206 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747300 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 22:04:58.749206 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747303 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 22:04:58.749206 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747305 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 22:04:58.749206 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747308 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 22:04:58.749206 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747311 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 22:04:58.749206 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747313 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 22:04:58.749206 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747315 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 22:04:58.749206 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747319 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 22:04:58.749206 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747323 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 22:04:58.749206 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747326 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 22:04:58.749206 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747329 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 22:04:58.749206 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747332 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 22:04:58.749206 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747335 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 22:04:58.749206 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.747338 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 22:04:58.749206 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748021 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 22:04:58.749206 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748029 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 22:04:58.749206 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748032 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 22:04:58.749206 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748035 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 22:04:58.749652 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748038 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 22:04:58.749652 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748041 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 22:04:58.749652 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748044 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 22:04:58.749652 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748046 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 22:04:58.749652 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748049 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 22:04:58.749652 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748052 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 22:04:58.749652 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748055 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 22:04:58.749652 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748057 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 22:04:58.749652 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748060 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 22:04:58.749652 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748062 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 22:04:58.749652 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748065 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 22:04:58.749652 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748067 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 22:04:58.749652 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748070 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 22:04:58.749652 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748073 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 22:04:58.749652 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748076 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 22:04:58.749652 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748079 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 22:04:58.749652 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748082 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 22:04:58.749652 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748084 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 22:04:58.749652 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748087 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 22:04:58.749652 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748089 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 22:04:58.750159 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748092 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 22:04:58.750159 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748094 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 22:04:58.750159 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748097 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 22:04:58.750159 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748099 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 22:04:58.750159 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748103 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 22:04:58.750159 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748106 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 22:04:58.750159 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748108 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 22:04:58.750159 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748113 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 22:04:58.750159 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748117 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 22:04:58.750159 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748120 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 22:04:58.750159 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748122 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 22:04:58.750159 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748125 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 22:04:58.750159 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748129 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 22:04:58.750159 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748132 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 22:04:58.750159 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748135 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 22:04:58.750159 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748138 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 22:04:58.750159 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748140 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 22:04:58.750159 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748143 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 22:04:58.750159 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748145 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 22:04:58.750159 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748148 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 22:04:58.750627 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748150 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 22:04:58.750627 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748153 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 22:04:58.750627 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748156 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 22:04:58.750627 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748158 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 22:04:58.750627 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748161 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 22:04:58.750627 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748163 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 22:04:58.750627 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748166 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 22:04:58.750627 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748169 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 22:04:58.750627 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748172 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 22:04:58.750627 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748175 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 22:04:58.750627 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748179 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 22:04:58.750627 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748183 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 22:04:58.750627 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748186 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 22:04:58.750627 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748190 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 22:04:58.750627 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748193 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 22:04:58.750627 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748196 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 22:04:58.750627 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748199 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 22:04:58.750627 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748202 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 22:04:58.750627 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748204 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 22:04:58.751094 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748208 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 22:04:58.751094 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748210 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 22:04:58.751094 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748213 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 22:04:58.751094 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748215 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 22:04:58.751094 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748217 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 22:04:58.751094 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748220 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 22:04:58.751094 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748223 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 22:04:58.751094 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748226 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 22:04:58.751094 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748229 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 22:04:58.751094 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748231 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 22:04:58.751094 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748234 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 22:04:58.751094 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748236 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 22:04:58.751094 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748239 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 22:04:58.751094 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748241 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 22:04:58.751094 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748244 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 22:04:58.751094 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748247 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 22:04:58.751094 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748249 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 22:04:58.751094 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748252 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 22:04:58.751094 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748254 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 22:04:58.751094 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748256 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 22:04:58.751594 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748260 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 22:04:58.751594 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748262 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 22:04:58.751594 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748265 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 22:04:58.751594 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748268 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 22:04:58.751594 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748270 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 22:04:58.751594 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748273 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 22:04:58.751594 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748276 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 22:04:58.751594 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748278 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 22:04:58.751594 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748280 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 22:04:58.751594 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748283 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 22:04:58.751594 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748285 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 22:04:58.751594 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748288 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 22:04:58.751594 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748290 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 22:04:58.751594 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748293 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 10 22:04:58.751594 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748295 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 22:04:58.751594 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748297 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 22:04:58.751594 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748649 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 22:04:58.751594 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748654 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 22:04:58.751594 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748656 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 22:04:58.752071 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748659 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 22:04:58.752071 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748662 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 22:04:58.752071 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748665 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 22:04:58.752071 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748668 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 22:04:58.752071 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748671 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 22:04:58.752071 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748674 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 22:04:58.752071 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748677 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 22:04:58.752071 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748680 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 22:04:58.752071 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748683 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 22:04:58.752071 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748686 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 22:04:58.752071 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748688 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 22:04:58.752071 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748691 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 22:04:58.752071 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748693 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 22:04:58.752071 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748697 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 22:04:58.752071 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748700 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 22:04:58.752071 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748703 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 22:04:58.752071 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748705 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 22:04:58.752071 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748708 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 22:04:58.752071 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748710 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 22:04:58.752071 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748713 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 10 22:04:58.752549 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748716 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 22:04:58.752549 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748718 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 22:04:58.752549 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748721 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 22:04:58.752549 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748723 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 22:04:58.752549 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748726 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 22:04:58.752549 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748728 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 22:04:58.752549 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748731 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 22:04:58.752549 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748733 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 22:04:58.752549 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748736 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 22:04:58.752549 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748738 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 22:04:58.752549 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748743 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 22:04:58.752549 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748746 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 22:04:58.752549 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748748 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 22:04:58.752549 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748751 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 22:04:58.752549 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748754 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 22:04:58.752549 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748757 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 22:04:58.752549 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748759 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 22:04:58.752549 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748762 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 22:04:58.752549 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748764 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 22:04:58.752549 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748767 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 22:04:58.753041 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748769 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 22:04:58.753041 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748772 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 22:04:58.753041 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748774 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 22:04:58.753041 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748777 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 22:04:58.753041 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748780 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 22:04:58.753041 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748782 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 22:04:58.753041 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748785 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 22:04:58.753041 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748788 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 22:04:58.753041 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748790 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 22:04:58.753041 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748793 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 22:04:58.753041 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748795 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 22:04:58.753041 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748797 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 22:04:58.753041 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748800 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 22:04:58.753041 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748802 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 22:04:58.753041 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748805 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 22:04:58.753041 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748807 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 22:04:58.753041 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748810 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 22:04:58.753041 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748812 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 22:04:58.753041 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748814 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 22:04:58.753041 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748817 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 22:04:58.753534 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748819 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 22:04:58.753534 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748822 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 22:04:58.753534 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748825 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 22:04:58.753534 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748828 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 22:04:58.753534 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748831 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 22:04:58.753534 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748835 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 22:04:58.753534 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748839 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 22:04:58.753534 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748842 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 22:04:58.753534 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748846 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 22:04:58.753534 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748848 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 22:04:58.753534 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748851 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 22:04:58.753534 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748854 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 22:04:58.753534 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748856 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 22:04:58.753534 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748859 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 22:04:58.753534 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748862 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 22:04:58.753534 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748864 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 22:04:58.753534 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748868 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 22:04:58.753534 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748871 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 22:04:58.753534 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748874 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 22:04:58.753999 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748878 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 22:04:58.753999 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748881 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 22:04:58.753999 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748884 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 22:04:58.753999 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748886 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 22:04:58.753999 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748889 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 22:04:58.753999 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748892 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 22:04:58.753999 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748895 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 22:04:58.753999 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748912 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 22:04:58.753999 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748916 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 22:04:58.753999 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748919 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 22:04:58.753999 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748921 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 22:04:58.753999 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748924 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 22:04:58.753999 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748927 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 22:04:58.753999 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748929 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 22:04:58.753999 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748932 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 22:04:58.753999 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748937 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 22:04:58.753999 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.748940 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 22:04:58.753999 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749016 2564 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 22:04:58.753999 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749050 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 22:04:58.753999 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749056 2564 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 22:04:58.753999 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749060 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 22:04:58.754528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749064 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 22:04:58.754528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749067 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 22:04:58.754528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749072 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 22:04:58.754528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749076 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 22:04:58.754528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749079 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 22:04:58.754528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749081 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 22:04:58.754528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749085 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 22:04:58.754528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749088 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 22:04:58.754528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749091 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 22:04:58.754528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749094 2564 flags.go:64] FLAG: --cgroup-root="" Sep 10 22:04:58.754528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749097 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 22:04:58.754528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749100 2564 flags.go:64] FLAG: --client-ca-file="" Sep 10 22:04:58.754528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749107 2564 flags.go:64] FLAG: --cloud-provider="external" Sep 10 22:04:58.754528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749110 2564 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 22:04:58.754528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749114 2564 flags.go:64] FLAG: --cluster-domain="" Sep 10 22:04:58.754528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749117 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 22:04:58.754528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749120 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 22:04:58.754528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749123 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 22:04:58.754528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749127 2564 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 22:04:58.754528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749130 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 22:04:58.754528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749133 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 22:04:58.754528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749137 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 22:04:58.754528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749140 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 22:04:58.755170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749143 2564 flags.go:64] FLAG: --contention-profiling="false" Sep 10 22:04:58.755170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749146 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 22:04:58.755170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749149 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 22:04:58.755170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749153 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 22:04:58.755170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749156 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 22:04:58.755170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749160 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 22:04:58.755170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749163 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 22:04:58.755170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749166 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 22:04:58.755170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749168 2564 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 22:04:58.755170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749172 2564 flags.go:64] FLAG: --enable-server="true" Sep 10 22:04:58.755170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749175 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 22:04:58.755170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749178 2564 flags.go:64] FLAG: --event-burst="100" Sep 10 22:04:58.755170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749181 2564 flags.go:64] FLAG: --event-qps="50" Sep 10 22:04:58.755170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749184 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 22:04:58.755170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749187 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 22:04:58.755170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749190 2564 flags.go:64] FLAG: --eviction-hard="" Sep 10 22:04:58.755170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749193 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 22:04:58.755170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749196 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 22:04:58.755170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749199 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 22:04:58.755170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749202 2564 flags.go:64] FLAG: --eviction-soft="" Sep 10 22:04:58.755170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749205 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 22:04:58.755170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749208 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 22:04:58.755170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749211 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 22:04:58.755170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749214 2564 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 22:04:58.755170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749217 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 22:04:58.756276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749220 2564 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 22:04:58.756276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749223 2564 flags.go:64] FLAG: --feature-gates="" Sep 10 22:04:58.756276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749226 2564 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 22:04:58.756276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749229 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 22:04:58.756276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749232 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 22:04:58.756276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749235 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 22:04:58.756276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749238 2564 flags.go:64] FLAG: --healthz-port="10248" Sep 10 22:04:58.756276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749241 2564 flags.go:64] FLAG: --help="false" Sep 10 22:04:58.756276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749244 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-137-148.ec2.internal" Sep 10 22:04:58.756276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749247 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 22:04:58.756276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749251 2564 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 22:04:58.756276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749253 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 22:04:58.756276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749256 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 22:04:58.756276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749260 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 22:04:58.756276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749263 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 22:04:58.756276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749265 2564 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 22:04:58.756276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749268 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 22:04:58.756276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749270 2564 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 22:04:58.756276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749273 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 22:04:58.756276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749276 2564 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 22:04:58.756276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749279 2564 flags.go:64] FLAG: --kube-reserved="" Sep 10 22:04:58.756276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749282 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 22:04:58.756276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749284 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 22:04:58.756276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749287 2564 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 22:04:58.757263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749292 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 22:04:58.757263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749295 2564 flags.go:64] FLAG: --lock-file="" Sep 10 22:04:58.757263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749298 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 22:04:58.757263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749300 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 22:04:58.757263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749303 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 22:04:58.757263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749308 2564 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 22:04:58.757263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749311 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 22:04:58.757263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749315 2564 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 22:04:58.757263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749318 2564 flags.go:64] FLAG: --logging-format="text" Sep 10 22:04:58.757263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749320 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 22:04:58.757263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749323 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 22:04:58.757263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749326 2564 flags.go:64] FLAG: --manifest-url="" Sep 10 22:04:58.757263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749329 2564 flags.go:64] FLAG: --manifest-url-header="" Sep 10 22:04:58.757263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749333 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 22:04:58.757263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749336 2564 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 22:04:58.757263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749339 2564 flags.go:64] FLAG: --max-pods="110" Sep 10 22:04:58.757263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749342 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 22:04:58.757263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749345 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 22:04:58.757263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749348 2564 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 22:04:58.757263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749351 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 22:04:58.757263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749354 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 22:04:58.757263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749357 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 22:04:58.757263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749361 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 22:04:58.757263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749368 2564 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 22:04:58.757263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749371 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 22:04:58.758179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749374 2564 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 22:04:58.758179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749377 2564 flags.go:64] FLAG: --pod-cidr="" Sep 10 22:04:58.758179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749379 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 10 22:04:58.758179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749384 2564 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 22:04:58.758179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749387 2564 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 22:04:58.758179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749390 2564 flags.go:64] FLAG: --pods-per-core="0" Sep 10 22:04:58.758179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749393 2564 flags.go:64] FLAG: --port="10250" Sep 10 22:04:58.758179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749400 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 22:04:58.758179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749403 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-077b882fd7b5156af" Sep 10 22:04:58.758179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749406 2564 flags.go:64] FLAG: --qos-reserved="" Sep 10 22:04:58.758179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749408 2564 flags.go:64] FLAG: --read-only-port="10255" Sep 10 22:04:58.758179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749411 2564 flags.go:64] FLAG: --register-node="true" Sep 10 22:04:58.758179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749414 2564 flags.go:64] FLAG: --register-with-taints="" Sep 10 22:04:58.758179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749418 2564 flags.go:64] FLAG: --registry-burst="10" Sep 10 22:04:58.758179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749421 2564 flags.go:64] FLAG: --registry-qps="5" Sep 10 22:04:58.758179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749424 2564 flags.go:64] FLAG: --reserved-cpus="" Sep 10 22:04:58.758179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749427 2564 flags.go:64] FLAG: --reserved-memory="" Sep 10 22:04:58.758179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749430 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 22:04:58.758179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749433 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 22:04:58.758179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749436 2564 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 22:04:58.758179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749439 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 22:04:58.758179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749442 2564 flags.go:64] FLAG: --runonce="false" Sep 10 22:04:58.758179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749445 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 22:04:58.758179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749448 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 22:04:58.758179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749451 2564 flags.go:64] FLAG: --seccomp-default="false" Sep 10 22:04:58.758775 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749454 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 22:04:58.758775 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749457 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 22:04:58.758775 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749460 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 22:04:58.758775 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749462 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 22:04:58.758775 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749465 2564 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 22:04:58.758775 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749468 2564 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 22:04:58.758775 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749471 2564 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 22:04:58.758775 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749473 2564 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 22:04:58.758775 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749476 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 22:04:58.758775 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749479 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 22:04:58.758775 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749482 2564 flags.go:64] FLAG: --system-cgroups="" Sep 10 22:04:58.758775 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749485 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 22:04:58.758775 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749490 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 22:04:58.758775 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749493 2564 flags.go:64] FLAG: --tls-cert-file="" Sep 10 22:04:58.758775 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749497 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 22:04:58.758775 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749524 2564 flags.go:64] FLAG: --tls-min-version="" Sep 10 22:04:58.758775 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749528 2564 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 22:04:58.758775 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749531 2564 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 22:04:58.758775 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749534 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 22:04:58.758775 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749537 2564 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 22:04:58.758775 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749540 2564 flags.go:64] FLAG: --v="2" Sep 10 22:04:58.758775 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749543 2564 flags.go:64] FLAG: --version="false" Sep 10 22:04:58.758775 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749551 2564 flags.go:64] FLAG: --vmodule="" Sep 10 22:04:58.758775 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749555 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 22:04:58.758775 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.749558 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 22:04:58.759387 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749675 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 22:04:58.759387 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749678 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 22:04:58.759387 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749682 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 22:04:58.759387 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749685 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 22:04:58.759387 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749687 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 22:04:58.759387 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749690 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 22:04:58.759387 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749692 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 22:04:58.759387 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749694 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 22:04:58.759387 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749697 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 22:04:58.759387 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749700 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 22:04:58.759387 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749702 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 22:04:58.759387 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749705 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 22:04:58.759387 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749709 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 22:04:58.759387 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749713 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 22:04:58.759387 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749716 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 22:04:58.759387 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749719 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 22:04:58.759387 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749722 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 22:04:58.759387 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749724 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 22:04:58.759387 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749727 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 22:04:58.759937 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749730 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 10 22:04:58.759937 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749732 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 22:04:58.759937 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749736 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 22:04:58.759937 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749739 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 22:04:58.759937 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749742 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 22:04:58.759937 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749744 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 22:04:58.759937 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749746 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 22:04:58.759937 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749749 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 22:04:58.759937 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749752 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 22:04:58.759937 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749755 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 22:04:58.759937 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749758 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 22:04:58.759937 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749761 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 22:04:58.759937 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749763 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 22:04:58.759937 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749766 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 22:04:58.759937 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749769 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 22:04:58.759937 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749771 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 22:04:58.759937 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749774 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 22:04:58.759937 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749776 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 22:04:58.759937 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749779 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 22:04:58.759937 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749781 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 22:04:58.760414 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749784 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 22:04:58.760414 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749787 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 22:04:58.760414 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749790 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 22:04:58.760414 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749793 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 22:04:58.760414 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749795 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 22:04:58.760414 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749799 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 22:04:58.760414 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749803 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 22:04:58.760414 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749806 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 22:04:58.760414 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749809 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 22:04:58.760414 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749811 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 22:04:58.760414 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749814 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 22:04:58.760414 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749817 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 22:04:58.760414 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749819 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 22:04:58.760414 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749822 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 22:04:58.760414 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749825 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 22:04:58.760414 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749828 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 22:04:58.760414 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749830 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 22:04:58.760414 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749833 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 22:04:58.760414 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749835 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 22:04:58.760414 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749838 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 22:04:58.760938 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749840 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 22:04:58.760938 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749843 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 22:04:58.760938 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749846 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 22:04:58.760938 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749848 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 22:04:58.760938 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749851 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 22:04:58.760938 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749854 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 22:04:58.760938 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749856 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 22:04:58.760938 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749859 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 22:04:58.760938 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749861 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 22:04:58.760938 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749864 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 22:04:58.760938 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749866 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 22:04:58.760938 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749869 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 22:04:58.760938 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749871 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 22:04:58.760938 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749875 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 22:04:58.760938 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749878 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 22:04:58.760938 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749881 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 22:04:58.760938 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749883 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 22:04:58.760938 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749886 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 22:04:58.760938 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749888 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 22:04:58.760938 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749891 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 22:04:58.761417 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749893 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 22:04:58.761417 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749909 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 22:04:58.761417 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749914 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 22:04:58.761417 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749918 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 22:04:58.761417 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749921 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 22:04:58.761417 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749925 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 22:04:58.761417 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749930 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 22:04:58.761417 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749932 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 22:04:58.761417 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749935 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 22:04:58.761417 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749938 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 22:04:58.761417 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749940 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 22:04:58.761417 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749944 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 22:04:58.761417 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749948 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 22:04:58.761417 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749951 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 22:04:58.761417 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749954 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 22:04:58.761417 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749957 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 22:04:58.761417 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749960 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 22:04:58.761417 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749963 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 22:04:58.761417 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749967 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 22:04:58.761870 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.749970 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 22:04:58.761870 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.750654 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 22:04:58.761870 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.758284 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 22:04:58.761870 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.758406 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 22:04:58.761870 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.759449 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 22:04:58.761870 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.759457 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 22:04:58.761870 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759522 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 22:04:58.761870 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759527 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 22:04:58.761870 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759531 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 22:04:58.761870 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759534 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 22:04:58.761870 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759537 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 22:04:58.761870 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759540 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 22:04:58.761870 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759543 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 22:04:58.761870 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759546 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 22:04:58.761870 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759549 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 22:04:58.762257 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759552 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 22:04:58.762257 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759555 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 22:04:58.762257 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759557 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 22:04:58.762257 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759560 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 22:04:58.762257 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759563 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 22:04:58.762257 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759567 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 22:04:58.762257 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759571 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 22:04:58.762257 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759574 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 10 22:04:58.762257 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759577 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 22:04:58.762257 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759580 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 22:04:58.762257 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759582 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 22:04:58.762257 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759585 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 22:04:58.762257 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759588 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 22:04:58.762257 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759590 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 22:04:58.762257 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759592 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 22:04:58.762257 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759596 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 22:04:58.762257 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759599 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 22:04:58.762257 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759601 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 22:04:58.762257 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759604 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 22:04:58.762257 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759607 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 22:04:58.762735 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759610 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 22:04:58.762735 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759613 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 22:04:58.762735 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759615 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 22:04:58.762735 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759618 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 22:04:58.762735 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759621 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 22:04:58.762735 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759623 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 22:04:58.762735 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759626 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 22:04:58.762735 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759628 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 22:04:58.762735 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759631 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 22:04:58.762735 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759633 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 22:04:58.762735 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759636 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 22:04:58.762735 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759639 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 22:04:58.762735 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759641 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 22:04:58.762735 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759645 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 22:04:58.762735 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759648 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 22:04:58.762735 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759651 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 22:04:58.762735 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759654 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 22:04:58.762735 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759657 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 22:04:58.762735 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759659 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 22:04:58.763199 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759662 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 22:04:58.763199 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759664 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 22:04:58.763199 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759667 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 22:04:58.763199 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759669 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 22:04:58.763199 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759672 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 22:04:58.763199 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759675 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 22:04:58.763199 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759677 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 22:04:58.763199 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759679 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 22:04:58.763199 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759682 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 22:04:58.763199 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759685 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 22:04:58.763199 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759687 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 22:04:58.763199 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759691 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 22:04:58.763199 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759693 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 22:04:58.763199 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759696 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 22:04:58.763199 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759699 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 22:04:58.763199 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759701 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 22:04:58.763199 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759704 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 22:04:58.763199 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759707 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 22:04:58.763199 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759709 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 22:04:58.763649 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759712 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 22:04:58.763649 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759714 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 22:04:58.763649 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759717 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 22:04:58.763649 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759719 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 22:04:58.763649 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759722 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 22:04:58.763649 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759726 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 22:04:58.763649 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759729 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 22:04:58.763649 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759732 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 22:04:58.763649 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759734 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 22:04:58.763649 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759737 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 22:04:58.763649 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759739 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 22:04:58.763649 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759742 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 22:04:58.763649 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759744 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 22:04:58.763649 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759747 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 22:04:58.763649 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759749 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 22:04:58.763649 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759752 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 22:04:58.763649 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759754 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 22:04:58.763649 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759756 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 22:04:58.763649 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759759 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 22:04:58.763649 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759761 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 22:04:58.764183 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759764 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 22:04:58.764183 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759766 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 22:04:58.764183 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759769 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 22:04:58.764183 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759772 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 22:04:58.764183 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759775 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 22:04:58.764183 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759778 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 22:04:58.764183 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759781 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 22:04:58.764183 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759784 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 22:04:58.764183 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759786 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 22:04:58.764183 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759789 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 22:04:58.764183 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759791 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 22:04:58.764183 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759794 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 22:04:58.764183 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.759798 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 22:04:58.764183 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759924 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 22:04:58.764183 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759930 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 22:04:58.764544 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759933 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 22:04:58.764544 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759936 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 22:04:58.764544 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759939 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 22:04:58.764544 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759942 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 22:04:58.764544 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759944 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 22:04:58.764544 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759947 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 22:04:58.764544 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759950 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 22:04:58.764544 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759953 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 22:04:58.764544 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759955 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 22:04:58.764544 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759958 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 22:04:58.764544 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759960 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 22:04:58.764544 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759963 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 22:04:58.764544 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759965 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 22:04:58.764544 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759968 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 22:04:58.764544 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759971 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 22:04:58.764544 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759974 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 22:04:58.764544 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759976 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 22:04:58.764544 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759978 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 22:04:58.764544 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759981 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 22:04:58.764996 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759983 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 22:04:58.764996 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759986 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 22:04:58.764996 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759990 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 22:04:58.764996 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759993 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 22:04:58.764996 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759996 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 22:04:58.764996 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.759999 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 22:04:58.764996 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760002 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 22:04:58.764996 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760004 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 22:04:58.764996 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760007 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 22:04:58.764996 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760009 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 22:04:58.764996 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760011 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 22:04:58.764996 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760014 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 22:04:58.764996 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760017 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 22:04:58.764996 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760019 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 22:04:58.764996 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760022 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 22:04:58.764996 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760024 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 22:04:58.764996 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760027 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 22:04:58.764996 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760029 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 22:04:58.764996 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760032 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 22:04:58.764996 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760034 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 22:04:58.765470 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760037 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 22:04:58.765470 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760039 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 22:04:58.765470 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760041 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 22:04:58.765470 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760044 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 10 22:04:58.765470 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760047 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 22:04:58.765470 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760049 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 22:04:58.765470 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760052 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 22:04:58.765470 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760055 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 22:04:58.765470 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760059 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 22:04:58.765470 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760063 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 22:04:58.765470 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760065 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 22:04:58.765470 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760068 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 22:04:58.765470 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760070 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 22:04:58.765470 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760073 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 22:04:58.765470 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760076 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 22:04:58.765470 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760079 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 22:04:58.765470 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760081 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 22:04:58.765470 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760084 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 22:04:58.765470 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760086 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 22:04:58.765470 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760089 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 22:04:58.765954 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760091 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 22:04:58.765954 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760093 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 22:04:58.765954 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760096 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 22:04:58.765954 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760098 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 22:04:58.765954 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760101 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 22:04:58.765954 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760103 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 22:04:58.765954 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760106 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 22:04:58.765954 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760110 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 22:04:58.765954 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760113 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 22:04:58.765954 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760116 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 22:04:58.765954 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760119 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 22:04:58.765954 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760121 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 22:04:58.765954 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760124 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 22:04:58.765954 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760126 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 22:04:58.765954 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760129 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 22:04:58.765954 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760131 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 22:04:58.765954 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760134 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 22:04:58.765954 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760136 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 22:04:58.765954 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760139 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 22:04:58.765954 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760141 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 22:04:58.766420 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760144 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 22:04:58.766420 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760146 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 22:04:58.766420 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760149 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 22:04:58.766420 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760151 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 22:04:58.766420 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760154 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 22:04:58.766420 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760156 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 22:04:58.766420 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760159 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 22:04:58.766420 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760162 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 22:04:58.766420 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760165 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 22:04:58.766420 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760167 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 22:04:58.766420 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760170 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 22:04:58.766420 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760172 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 22:04:58.766420 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760175 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 22:04:58.766420 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760179 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 22:04:58.766420 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760182 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 22:04:58.766420 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760184 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 22:04:58.766420 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760188 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 22:04:58.766420 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:04:58.760191 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 22:04:58.766839 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.760196 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 22:04:58.766839 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.760291 2564 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 22:04:58.766839 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.765303 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 22:04:58.766839 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.766347 2564 server.go:1019] "Starting client certificate rotation" Sep 10 22:04:58.766839 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.766450 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 22:04:58.766839 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.766487 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 22:04:58.769170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.769158 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 22:04:58.770344 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.770326 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 22:04:58.779969 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.779949 2564 log.go:25] "Validated CRI v1 runtime API" Sep 10 22:04:58.784006 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.783992 2564 log.go:25] "Validated CRI v1 image API" Sep 10 22:04:58.784854 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.784839 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 22:04:58.790622 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.790605 2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b1931cf6-acd7-4e80-81d7-8e41993b9ee9:/dev/nvme0n1p3 e6e0329d-dfcd-4397-9b75-bfd7fde19651:/dev/nvme0n1p4] Sep 10 22:04:58.790685 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.790623 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 22:04:58.795622 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.795516 2564 manager.go:216] Machine: {Timestamp:2026-09-10 22:04:58.794116046 +0000 UTC m=+0.363606001 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200225 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2081270f113d6837c74717d6fd48f8 SystemUUID:ec208127-0f11-3d68-37c7-4717d6fd48f8 BootID:35f3dd1f-2c30-4238-a5e3-07d3f9442c06 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5b:2a:ff:b0:a1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5b:2a:ff:b0:a1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:0e:ef:8c:dd:68 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 22:04:58.795622 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.795616 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 22:04:58.795727 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.795693 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 22:04:58.798085 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.798069 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 22:04:58.799448 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.799422 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 22:04:58.799610 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.799451 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-148.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 22:04:58.801156 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.801146 2564 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 22:04:58.801189 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.801158 2564 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 22:04:58.801189 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.801166 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 22:04:58.801956 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.801946 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 22:04:58.803163 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.803154 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 22:04:58.807736 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.807725 2564 state_mem.go:36] "Initialized new in-memory state store" Sep 10 22:04:58.807853 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.807845 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 22:04:58.807887 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.807878 2564 kubelet.go:486] "Attempting to sync node with API server" Sep 10 22:04:58.807941 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.807889 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 22:04:58.807941 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.807913 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 22:04:58.807941 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.807923 2564 kubelet.go:398] "Adding apiserver pod source" Sep 10 22:04:58.808056 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.807950 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 22:04:58.808279 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.808268 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 22:04:58.808325 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.808286 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 22:04:58.811362 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.811346 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 22:04:58.814203 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.814190 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 22:04:58.814248 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.814219 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 22:04:58.816340 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.816327 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 22:04:58.816388 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.816349 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 22:04:58.816388 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.816357 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 22:04:58.816388 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.816366 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 22:04:58.816388 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.816379 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 22:04:58.816388 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.816386 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 22:04:58.816511 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.816424 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 22:04:58.816511 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.816433 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 22:04:58.816511 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.816444 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 22:04:58.816511 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.816454 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 22:04:58.816511 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.816474 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 22:04:58.816511 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.816482 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 22:04:58.817399 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.817388 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 22:04:58.817437 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.817400 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 22:04:58.818396 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:58.818340 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 22:04:58.818449 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:58.818422 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-148.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 22:04:58.821144 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.821131 2564 server.go:1268] "Started kubelet" Sep 10 22:04:58.821218 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.821196 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 22:04:58.825600 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.821222 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 22:04:58.825699 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.825635 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 22:04:58.826299 ip-10-0-137-148 systemd[1]: Started Kubernetes Kubelet. Sep 10 22:04:58.827292 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.827230 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 22:04:58.830764 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.830751 2564 server.go:310] "Adding debug handlers to kubelet server" Sep 10 22:04:58.832935 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:58.832826 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 22:04:58.834527 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:58.833461 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-148.ec2.internal.18d414debabbe393 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-148.ec2.internal,UID:ip-10-0-137-148.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-148.ec2.internal,},FirstTimestamp:2026-09-10 22:04:58.821100435 +0000 UTC m=+0.390590391,LastTimestamp:2026-09-10 22:04:58.821100435 +0000 UTC m=+0.390590391,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-148.ec2.internal,}" Sep 10 22:04:58.838212 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.838195 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 22:04:58.838875 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.838843 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 22:04:58.839183 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:58.839129 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-148.ec2.internal\" not found" Sep 10 22:04:58.839716 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.839700 2564 factory.go:55] Registering systemd factory Sep 10 22:04:58.839791 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.839740 2564 factory.go:223] Registration of the systemd container factory successfully Sep 10 22:04:58.839791 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.839759 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 22:04:58.839791 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.839759 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 22:04:58.839791 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.839786 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 22:04:58.839983 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.839888 2564 reconstruct.go:97] "Volume reconstruction finished" Sep 10 22:04:58.839983 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.839913 2564 reconciler.go:29] "Reconciler: start to sync state" Sep 10 22:04:58.839983 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.839962 2564 factory.go:153] Registering CRI-O factory Sep 10 22:04:58.839983 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.839974 2564 factory.go:223] Registration of the crio container factory successfully Sep 10 22:04:58.840099 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.840027 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 22:04:58.840099 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.840046 2564 factory.go:103] Registering Raw factory Sep 10 22:04:58.840099 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.840061 2564 manager.go:1196] Started watching for new ooms in manager Sep 10 22:04:58.840522 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.840506 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-148.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 22:04:58.840522 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.840519 2564 manager.go:349] Starting recovery of all containers Sep 10 22:04:58.844528 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:58.844502 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-148.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 22:04:58.844528 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:58.844502 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 22:04:58.850894 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.850877 2564 manager.go:354] Recovery completed Sep 10 22:04:58.855208 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.855195 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 22:04:58.857605 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.857565 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientMemory" Sep 10 22:04:58.857740 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.857727 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 22:04:58.857824 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.857815 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientPID" Sep 10 22:04:58.859248 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.859227 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-grs6s" Sep 10 22:04:58.859603 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.859583 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 22:04:58.859603 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.859599 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 22:04:58.859743 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.859614 2564 state_mem.go:36] "Initialized new in-memory state store" Sep 10 22:04:58.859743 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.859631 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 22:04:58.859889 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:58.859819 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-148.ec2.internal.18d414debcea7dfe default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-148.ec2.internal,UID:ip-10-0-137-148.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-148.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-148.ec2.internal,},FirstTimestamp:2026-09-10 22:04:58.857709054 +0000 UTC m=+0.427199009,LastTimestamp:2026-09-10 22:04:58.857709054 +0000 UTC m=+0.427199009,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-148.ec2.internal,}" Sep 10 22:04:58.860494 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.860476 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientMemory" Sep 10 22:04:58.860572 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.860510 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 22:04:58.860572 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.860523 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientPID" Sep 10 22:04:58.861887 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.861875 2564 policy_none.go:49] "None policy: Start" Sep 10 22:04:58.861887 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.861890 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 22:04:58.861993 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.861923 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 22:04:58.863937 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.863909 2564 policy_none.go:47] "Start" Sep 10 22:04:58.866567 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.866549 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-grs6s" Sep 10 22:04:58.868142 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:58.868034 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-148.ec2.internal.18d414debcebe938 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-148.ec2.internal,UID:ip-10-0-137-148.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-148.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-148.ec2.internal,},FirstTimestamp:2026-09-10 22:04:58.85780204 +0000 UTC m=+0.427291999,LastTimestamp:2026-09-10 22:04:58.85780204 +0000 UTC m=+0.427291999,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-148.ec2.internal,}" Sep 10 22:04:58.870803 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:58.870730 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-148.ec2.internal.18d414debced2c3a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-148.ec2.internal,UID:ip-10-0-137-148.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-148.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-148.ec2.internal,},FirstTimestamp:2026-09-10 22:04:58.85788473 +0000 UTC m=+0.427374689,LastTimestamp:2026-09-10 22:04:58.85788473 +0000 UTC m=+0.427374689,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-148.ec2.internal,}" Sep 10 22:04:58.908348 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.901420 2564 manager.go:337] "Starting Device Plugin manager" Sep 10 22:04:58.908348 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:58.901452 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 22:04:58.908348 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.901462 2564 server.go:87] "Starting device plugin registration server" Sep 10 22:04:58.908348 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.901830 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 22:04:58.908348 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.901841 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 22:04:58.908348 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.901965 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 22:04:58.908348 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.902040 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 22:04:58.908348 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.902046 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 22:04:58.908348 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:58.902519 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 22:04:58.908348 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:58.902585 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-148.ec2.internal\" not found" Sep 10 22:04:58.929382 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.929355 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 22:04:58.930437 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.930414 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 22:04:58.930507 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.930476 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 22:04:58.930507 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.930503 2564 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 22:04:58.930595 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:58.930545 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 22:04:58.932456 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:58.932434 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 22:04:59.002222 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.002169 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 22:04:59.004026 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.004008 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientMemory" Sep 10 22:04:59.004123 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.004038 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 22:04:59.004123 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.004052 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientPID" Sep 10 22:04:59.004123 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.004081 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-148.ec2.internal" Sep 10 22:04:59.012085 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.012068 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-148.ec2.internal" Sep 10 22:04:59.012140 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:59.012091 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-148.ec2.internal\": node \"ip-10-0-137-148.ec2.internal\" not found" Sep 10 22:04:59.031606 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.031587 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-148.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal"] Sep 10 22:04:59.031680 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.031655 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 22:04:59.033057 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.033041 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientMemory" Sep 10 22:04:59.033148 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.033069 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 22:04:59.033148 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.033079 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientPID" Sep 10 22:04:59.035361 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.035350 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 22:04:59.035473 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.035459 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.ec2.internal" Sep 10 22:04:59.035530 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.035491 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 22:04:59.036035 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.036018 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientMemory" Sep 10 22:04:59.036119 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.036049 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 22:04:59.036119 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.036024 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientMemory" Sep 10 22:04:59.036119 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.036064 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientPID" Sep 10 22:04:59.036119 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.036084 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 22:04:59.036119 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.036100 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientPID" Sep 10 22:04:59.038369 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.038350 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" Sep 10 22:04:59.038450 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.038382 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 22:04:59.039078 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.039064 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientMemory" Sep 10 22:04:59.039157 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.039088 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 22:04:59.039157 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.039099 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeHasSufficientPID" Sep 10 22:04:59.041242 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:59.041173 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-148.ec2.internal\" not found" Sep 10 22:04:59.070669 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:59.070650 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-148.ec2.internal\" not found" node="ip-10-0-137-148.ec2.internal" Sep 10 22:04:59.074854 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:59.074839 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-148.ec2.internal\" not found" node="ip-10-0-137-148.ec2.internal" Sep 10 22:04:59.140337 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.140315 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45d26a6c9bf1e3308cfe50ca1c9f6711-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal\" (UID: \"45d26a6c9bf1e3308cfe50ca1c9f6711\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" Sep 10 22:04:59.140440 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.140340 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45d26a6c9bf1e3308cfe50ca1c9f6711-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal\" (UID: \"45d26a6c9bf1e3308cfe50ca1c9f6711\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" Sep 10 22:04:59.140440 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.140370 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f09bb482ad5b644a06b2ecf966b25fc2-config\") pod \"kube-apiserver-proxy-ip-10-0-137-148.ec2.internal\" (UID: \"f09bb482ad5b644a06b2ecf966b25fc2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.ec2.internal" Sep 10 22:04:59.141340 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:59.141327 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-148.ec2.internal\" not found" Sep 10 22:04:59.241107 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.241088 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f09bb482ad5b644a06b2ecf966b25fc2-config\") pod \"kube-apiserver-proxy-ip-10-0-137-148.ec2.internal\" (UID: \"f09bb482ad5b644a06b2ecf966b25fc2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.ec2.internal" Sep 10 22:04:59.241223 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.241111 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45d26a6c9bf1e3308cfe50ca1c9f6711-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal\" (UID: \"45d26a6c9bf1e3308cfe50ca1c9f6711\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" Sep 10 22:04:59.241223 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.241130 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45d26a6c9bf1e3308cfe50ca1c9f6711-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal\" (UID: \"45d26a6c9bf1e3308cfe50ca1c9f6711\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" Sep 10 22:04:59.241223 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.241184 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45d26a6c9bf1e3308cfe50ca1c9f6711-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal\" (UID: \"45d26a6c9bf1e3308cfe50ca1c9f6711\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" Sep 10 22:04:59.241223 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.241196 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f09bb482ad5b644a06b2ecf966b25fc2-config\") pod \"kube-apiserver-proxy-ip-10-0-137-148.ec2.internal\" (UID: \"f09bb482ad5b644a06b2ecf966b25fc2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.ec2.internal" Sep 10 22:04:59.241223 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.241200 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45d26a6c9bf1e3308cfe50ca1c9f6711-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal\" (UID: \"45d26a6c9bf1e3308cfe50ca1c9f6711\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" Sep 10 22:04:59.242129 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:59.242113 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-148.ec2.internal\" not found" Sep 10 22:04:59.342698 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:59.342631 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-148.ec2.internal\" not found" Sep 10 22:04:59.373851 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.373822 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.ec2.internal" Sep 10 22:04:59.377597 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.377579 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" Sep 10 22:04:59.442833 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:59.442795 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-148.ec2.internal\" not found" Sep 10 22:04:59.543861 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:59.543820 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-148.ec2.internal\" not found" Sep 10 22:04:59.644998 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:59.644880 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-148.ec2.internal\" not found" Sep 10 22:04:59.737042 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.737015 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 22:04:59.739990 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.739960 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.ec2.internal" Sep 10 22:04:59.750823 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.750794 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 22:04:59.752426 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.752411 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" Sep 10 22:04:59.765907 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.765879 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 22:04:59.766065 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.766046 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 22:04:59.766134 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.766107 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 22:04:59.766181 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:59.766116 2564 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://af8eee50ce7be4940a082bf9554c4827-9e5117b1a4dca35e.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.137.148:34336->52.207.84.252:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" Sep 10 22:04:59.810528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.810482 2564 apiserver.go:52] "Watching apiserver" Sep 10 22:04:59.827237 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.827208 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 22:04:59.830071 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.829549 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg","openshift-cluster-node-tuning-operator/tuned-8gs2q","openshift-dns/node-resolver-zgwrg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal","openshift-multus/multus-additional-cni-plugins-lzxzf","openshift-image-registry/node-ca-ph2dv","openshift-multus/multus-95n7s","openshift-multus/network-metrics-daemon-fp72v","openshift-network-diagnostics/network-check-target-wfb5h","openshift-network-operator/iptables-alerter-nx756","openshift-ovn-kubernetes/ovnkube-node-5j226","kube-system/global-pull-secret-syncer-m6nqn","kube-system/konnectivity-agent-96t46","kube-system/kube-apiserver-proxy-ip-10-0-137-148.ec2.internal"] Sep 10 22:04:59.832459 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.832434 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nx756" Sep 10 22:04:59.834807 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.834784 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.835386 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.835362 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 22:04:59.835386 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.835365 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 22:04:59.835507 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.835482 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2cbt4\"" Sep 10 22:04:59.835554 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.835510 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 22:04:59.837066 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.837047 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zgwrg" Sep 10 22:04:59.837757 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.837736 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 22:04:59.837757 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.837755 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 22:04:59.837867 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.837754 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 22:04:59.837867 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.837825 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 22:04:59.838335 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.838318 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 22:04:59.838964 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.838949 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 22:04:59.838964 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.838953 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 22:04:59.839067 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.838954 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qsmdd\"" Sep 10 22:04:59.839473 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.839454 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:04:59.839548 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.839498 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cjlzl\"" Sep 10 22:04:59.839548 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.839519 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 22:04:59.839548 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:59.839522 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp72v" podUID="d748c035-3205-4b2f-8b7e-a229ff50e466" Sep 10 22:04:59.839669 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.839501 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 22:04:59.841715 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.841697 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lzxzf" Sep 10 22:04:59.843982 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.843964 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ph2dv" Sep 10 22:04:59.844306 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844283 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 22:04:59.844405 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844287 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.844405 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844343 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 22:04:59.844405 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844343 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nplk2\"" Sep 10 22:04:59.844565 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844345 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-ovn-node-metrics-cert\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.844565 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844473 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 22:04:59.844565 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844485 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-node-log\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.844565 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844536 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0638522d-7b41-42c0-a93c-61fe7e00935b-tmp-dir\") pod \"node-resolver-zgwrg\" (UID: \"0638522d-7b41-42c0-a93c-61fe7e00935b\") " pod="openshift-dns/node-resolver-zgwrg" Sep 10 22:04:59.844767 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844565 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02564ad8-b933-4bc4-9e94-8995c093b6b0-cni-binary-copy\") pod \"multus-additional-cni-plugins-lzxzf\" (UID: \"02564ad8-b933-4bc4-9e94-8995c093b6b0\") " pod="openshift-multus/multus-additional-cni-plugins-lzxzf" Sep 10 22:04:59.844767 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844593 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/02564ad8-b933-4bc4-9e94-8995c093b6b0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lzxzf\" (UID: \"02564ad8-b933-4bc4-9e94-8995c093b6b0\") " pod="openshift-multus/multus-additional-cni-plugins-lzxzf" Sep 10 22:04:59.844767 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844612 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 22:04:59.844767 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844627 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctzb2\" (UniqueName: \"kubernetes.io/projected/02564ad8-b933-4bc4-9e94-8995c093b6b0-kube-api-access-ctzb2\") pod \"multus-additional-cni-plugins-lzxzf\" (UID: \"02564ad8-b933-4bc4-9e94-8995c093b6b0\") " pod="openshift-multus/multus-additional-cni-plugins-lzxzf" Sep 10 22:04:59.844767 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844651 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-etc-openvswitch\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.844767 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844682 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-host-run-ovn-kubernetes\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.844767 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844717 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0638522d-7b41-42c0-a93c-61fe7e00935b-hosts-file\") pod \"node-resolver-zgwrg\" (UID: \"0638522d-7b41-42c0-a93c-61fe7e00935b\") " pod="openshift-dns/node-resolver-zgwrg" Sep 10 22:04:59.844767 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844754 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-host-kubelet\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.844767 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844771 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-host-slash\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.845186 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844786 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-log-socket\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.845186 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844802 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-ovnkube-config\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.845186 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844826 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02564ad8-b933-4bc4-9e94-8995c093b6b0-system-cni-dir\") pod \"multus-additional-cni-plugins-lzxzf\" (UID: \"02564ad8-b933-4bc4-9e94-8995c093b6b0\") " pod="openshift-multus/multus-additional-cni-plugins-lzxzf" Sep 10 22:04:59.845186 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844843 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02564ad8-b933-4bc4-9e94-8995c093b6b0-cnibin\") pod \"multus-additional-cni-plugins-lzxzf\" (UID: \"02564ad8-b933-4bc4-9e94-8995c093b6b0\") " pod="openshift-multus/multus-additional-cni-plugins-lzxzf" Sep 10 22:04:59.845186 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844859 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tbgb\" (UniqueName: \"kubernetes.io/projected/5d91688c-bfc4-4fea-90c5-0c45ebfab0b5-kube-api-access-8tbgb\") pod \"iptables-alerter-nx756\" (UID: \"5d91688c-bfc4-4fea-90c5-0c45ebfab0b5\") " pod="openshift-network-operator/iptables-alerter-nx756" Sep 10 22:04:59.845186 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844876 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-env-overrides\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.845186 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844928 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-ovnkube-script-lib\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.845186 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844950 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02564ad8-b933-4bc4-9e94-8995c093b6b0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lzxzf\" (UID: \"02564ad8-b933-4bc4-9e94-8995c093b6b0\") " pod="openshift-multus/multus-additional-cni-plugins-lzxzf" Sep 10 22:04:59.845186 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844964 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5d91688c-bfc4-4fea-90c5-0c45ebfab0b5-host-slash\") pod \"iptables-alerter-nx756\" (UID: \"5d91688c-bfc4-4fea-90c5-0c45ebfab0b5\") " pod="openshift-network-operator/iptables-alerter-nx756" Sep 10 22:04:59.845186 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.844984 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-var-lib-openvswitch\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.845186 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.845021 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-host-cni-netd\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.845186 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.845047 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95rqt\" (UniqueName: \"kubernetes.io/projected/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-kube-api-access-95rqt\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.845186 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.845067 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tl88\" (UniqueName: \"kubernetes.io/projected/0638522d-7b41-42c0-a93c-61fe7e00935b-kube-api-access-6tl88\") pod \"node-resolver-zgwrg\" (UID: \"0638522d-7b41-42c0-a93c-61fe7e00935b\") " pod="openshift-dns/node-resolver-zgwrg" Sep 10 22:04:59.845186 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.845090 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs\") pod \"network-metrics-daemon-fp72v\" (UID: \"d748c035-3205-4b2f-8b7e-a229ff50e466\") " pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:04:59.845186 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.845108 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-run-ovn\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.845186 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.845122 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-host-cni-bin\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.845723 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.845143 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82j6p\" (UniqueName: \"kubernetes.io/projected/d748c035-3205-4b2f-8b7e-a229ff50e466-kube-api-access-82j6p\") pod \"network-metrics-daemon-fp72v\" (UID: \"d748c035-3205-4b2f-8b7e-a229ff50e466\") " pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:04:59.845723 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.845176 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/02564ad8-b933-4bc4-9e94-8995c093b6b0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lzxzf\" (UID: \"02564ad8-b933-4bc4-9e94-8995c093b6b0\") " pod="openshift-multus/multus-additional-cni-plugins-lzxzf" Sep 10 22:04:59.845723 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.845202 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5d91688c-bfc4-4fea-90c5-0c45ebfab0b5-iptables-alerter-script\") pod \"iptables-alerter-nx756\" (UID: \"5d91688c-bfc4-4fea-90c5-0c45ebfab0b5\") " pod="openshift-network-operator/iptables-alerter-nx756" Sep 10 22:04:59.845723 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.845218 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-systemd-units\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.845723 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.845250 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-run-openvswitch\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.845723 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.845269 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-host-run-netns\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.845723 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.845283 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-run-systemd\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.845927 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.845862 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 22:04:59.846400 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.846381 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 22:04:59.846454 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.846416 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gtppr\"" Sep 10 22:04:59.846539 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.846516 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:04:59.846630 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:59.846588 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wfb5h" podUID="493e7732-8863-4f0f-b93e-bce47d8cf3de" Sep 10 22:04:59.847200 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.847156 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 22:04:59.847465 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.847444 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 22:04:59.848902 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.848885 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-95n7s" Sep 10 22:04:59.851320 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.851302 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sfgwp\"" Sep 10 22:04:59.851398 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.851325 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:04:59.851398 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.851344 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 22:04:59.852661 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.852643 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 22:04:59.853898 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.853880 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-89pn5\"" Sep 10 22:04:59.853992 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.853975 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" Sep 10 22:04:59.854191 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.854166 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 22:04:59.854294 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.854205 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 22:04:59.855988 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.855961 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 22:04:59.856523 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.856507 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 22:04:59.856595 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.856578 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 22:04:59.856595 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.856579 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 22:04:59.856698 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.856682 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2zdlw\"" Sep 10 22:04:59.856847 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.856829 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-96t46" Sep 10 22:04:59.859392 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.859371 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:04:59.859481 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:59.859455 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6nqn" podUID="673499d4-211d-40e2-8a3f-95c74bc1d0f3" Sep 10 22:04:59.859911 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.859893 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 22:04:59.860041 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.859972 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pbdpp\"" Sep 10 22:04:59.860352 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.860336 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 22:04:59.872404 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.872355 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 21:59:59 +0000 UTC" deadline="2028-03-22 02:22:15.290886847 +0000 UTC" Sep 10 22:04:59.872404 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.872404 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13396h17m15.418487707s" Sep 10 22:04:59.881837 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.881811 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-db99r" Sep 10 22:04:59.893236 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.893210 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-db99r" Sep 10 22:04:59.941046 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.941025 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 22:04:59.946172 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946153 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02564ad8-b933-4bc4-9e94-8995c093b6b0-system-cni-dir\") pod \"multus-additional-cni-plugins-lzxzf\" (UID: \"02564ad8-b933-4bc4-9e94-8995c093b6b0\") " pod="openshift-multus/multus-additional-cni-plugins-lzxzf" Sep 10 22:04:59.946232 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946181 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02564ad8-b933-4bc4-9e94-8995c093b6b0-cnibin\") pod \"multus-additional-cni-plugins-lzxzf\" (UID: \"02564ad8-b933-4bc4-9e94-8995c093b6b0\") " pod="openshift-multus/multus-additional-cni-plugins-lzxzf" Sep 10 22:04:59.946232 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946201 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02564ad8-b933-4bc4-9e94-8995c093b6b0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lzxzf\" (UID: \"02564ad8-b933-4bc4-9e94-8995c093b6b0\") " pod="openshift-multus/multus-additional-cni-plugins-lzxzf" Sep 10 22:04:59.946232 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946219 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-var-lib-openvswitch\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.946364 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946247 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6d6ce980-2a0b-474d-afe7-a72bd2e04f57-device-dir\") pod \"aws-ebs-csi-driver-node-7lbpg\" (UID: \"6d6ce980-2a0b-474d-afe7-a72bd2e04f57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" Sep 10 22:04:59.946364 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946264 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02564ad8-b933-4bc4-9e94-8995c093b6b0-system-cni-dir\") pod \"multus-additional-cni-plugins-lzxzf\" (UID: \"02564ad8-b933-4bc4-9e94-8995c093b6b0\") " pod="openshift-multus/multus-additional-cni-plugins-lzxzf" Sep 10 22:04:59.946364 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946269 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02564ad8-b933-4bc4-9e94-8995c093b6b0-cnibin\") pod \"multus-additional-cni-plugins-lzxzf\" (UID: \"02564ad8-b933-4bc4-9e94-8995c093b6b0\") " pod="openshift-multus/multus-additional-cni-plugins-lzxzf" Sep 10 22:04:59.946364 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946273 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6d6ce980-2a0b-474d-afe7-a72bd2e04f57-etc-selinux\") pod \"aws-ebs-csi-driver-node-7lbpg\" (UID: \"6d6ce980-2a0b-474d-afe7-a72bd2e04f57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" Sep 10 22:04:59.946364 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946309 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-var-lib-openvswitch\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.946364 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946329 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4f2170c9-cacb-47f3-8a42-b80039deefef-agent-certs\") pod \"konnectivity-agent-96t46\" (UID: \"4f2170c9-cacb-47f3-8a42-b80039deefef\") " pod="kube-system/konnectivity-agent-96t46" Sep 10 22:04:59.946364 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946351 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs\") pod \"network-metrics-daemon-fp72v\" (UID: \"d748c035-3205-4b2f-8b7e-a229ff50e466\") " pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:04:59.946568 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946377 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-run-ovn\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.946568 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946394 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-host-cni-bin\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.946568 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946410 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7114320-6f0b-4cc6-85a8-309ddda645af-host\") pod \"node-ca-ph2dv\" (UID: \"c7114320-6f0b-4cc6-85a8-309ddda645af\") " pod="openshift-image-registry/node-ca-ph2dv" Sep 10 22:04:59.946568 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946425 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-system-cni-dir\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:04:59.946568 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946432 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-run-ovn\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.946568 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946387 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02564ad8-b933-4bc4-9e94-8995c093b6b0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lzxzf\" (UID: \"02564ad8-b933-4bc4-9e94-8995c093b6b0\") " pod="openshift-multus/multus-additional-cni-plugins-lzxzf" Sep 10 22:04:59.946568 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946439 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/039f078c-f211-4f4f-8878-d39a9e529672-cni-binary-copy\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:04:59.946568 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946436 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-host-cni-bin\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.946568 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:59.946475 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 22:04:59.946568 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946488 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-host-var-lib-kubelet\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:04:59.946568 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946506 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6d6ce980-2a0b-474d-afe7-a72bd2e04f57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7lbpg\" (UID: \"6d6ce980-2a0b-474d-afe7-a72bd2e04f57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" Sep 10 22:04:59.946568 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946535 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/02564ad8-b933-4bc4-9e94-8995c093b6b0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lzxzf\" (UID: \"02564ad8-b933-4bc4-9e94-8995c093b6b0\") " pod="openshift-multus/multus-additional-cni-plugins-lzxzf" Sep 10 22:04:59.946568 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:04:59.946562 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs podName:d748c035-3205-4b2f-8b7e-a229ff50e466 nodeName:}" failed. No retries permitted until 2026-09-10 22:05:00.446530746 +0000 UTC m=+2.016020695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs") pod "network-metrics-daemon-fp72v" (UID: "d748c035-3205-4b2f-8b7e-a229ff50e466") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 22:04:59.946950 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946618 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-systemd-units\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.946950 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946645 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6d6ce980-2a0b-474d-afe7-a72bd2e04f57-registration-dir\") pod \"aws-ebs-csi-driver-node-7lbpg\" (UID: \"6d6ce980-2a0b-474d-afe7-a72bd2e04f57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" Sep 10 22:04:59.946950 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946663 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-run\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:04:59.946950 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946678 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-host\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:04:59.946950 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946714 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b077b17c-710b-474e-906e-8f86682b86a2-tmp\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:04:59.946950 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946725 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-systemd-units\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.946950 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946734 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-run-systemd\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.946950 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946774 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-run-systemd\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.946950 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946812 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.946950 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946842 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-host-run-multus-certs\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:04:59.946950 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946867 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.946950 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946868 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0638522d-7b41-42c0-a93c-61fe7e00935b-tmp-dir\") pod \"node-resolver-zgwrg\" (UID: \"0638522d-7b41-42c0-a93c-61fe7e00935b\") " pod="openshift-dns/node-resolver-zgwrg" Sep 10 22:04:59.946950 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946908 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02564ad8-b933-4bc4-9e94-8995c093b6b0-cni-binary-copy\") pod \"multus-additional-cni-plugins-lzxzf\" (UID: \"02564ad8-b933-4bc4-9e94-8995c093b6b0\") " pod="openshift-multus/multus-additional-cni-plugins-lzxzf" Sep 10 22:04:59.947438 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946959 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/673499d4-211d-40e2-8a3f-95c74bc1d0f3-kubelet-config\") pod \"global-pull-secret-syncer-m6nqn\" (UID: \"673499d4-211d-40e2-8a3f-95c74bc1d0f3\") " pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:04:59.947438 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.946985 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-etc-sysctl-d\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:04:59.947438 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947008 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-etc-systemd\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:04:59.947438 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947032 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-lib-modules\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:04:59.947438 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947058 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-765sx\" (UniqueName: \"kubernetes.io/projected/493e7732-8863-4f0f-b93e-bce47d8cf3de-kube-api-access-765sx\") pod \"network-check-target-wfb5h\" (UID: \"493e7732-8863-4f0f-b93e-bce47d8cf3de\") " pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:04:59.947438 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947099 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-host-run-netns\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:04:59.947438 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947123 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scp8w\" (UniqueName: \"kubernetes.io/projected/039f078c-f211-4f4f-8878-d39a9e529672-kube-api-access-scp8w\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:04:59.947438 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947124 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/02564ad8-b933-4bc4-9e94-8995c093b6b0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lzxzf\" (UID: \"02564ad8-b933-4bc4-9e94-8995c093b6b0\") " pod="openshift-multus/multus-additional-cni-plugins-lzxzf" Sep 10 22:04:59.947438 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947147 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zs5s\" (UniqueName: \"kubernetes.io/projected/6d6ce980-2a0b-474d-afe7-a72bd2e04f57-kube-api-access-4zs5s\") pod \"aws-ebs-csi-driver-node-7lbpg\" (UID: \"6d6ce980-2a0b-474d-afe7-a72bd2e04f57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" Sep 10 22:04:59.947438 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947174 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4f2170c9-cacb-47f3-8a42-b80039deefef-konnectivity-ca\") pod \"konnectivity-agent-96t46\" (UID: \"4f2170c9-cacb-47f3-8a42-b80039deefef\") " pod="kube-system/konnectivity-agent-96t46" Sep 10 22:04:59.947438 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947184 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0638522d-7b41-42c0-a93c-61fe7e00935b-tmp-dir\") pod \"node-resolver-zgwrg\" (UID: \"0638522d-7b41-42c0-a93c-61fe7e00935b\") " pod="openshift-dns/node-resolver-zgwrg" Sep 10 22:04:59.947438 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947209 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-log-socket\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.947438 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947234 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-ovnkube-config\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.947438 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947274 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-multus-cni-dir\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:04:59.947438 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947273 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-log-socket\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.947438 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947295 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-cnibin\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:04:59.947438 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947328 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-host-var-lib-cni-multus\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:04:59.948209 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947349 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tbgb\" (UniqueName: \"kubernetes.io/projected/5d91688c-bfc4-4fea-90c5-0c45ebfab0b5-kube-api-access-8tbgb\") pod \"iptables-alerter-nx756\" (UID: \"5d91688c-bfc4-4fea-90c5-0c45ebfab0b5\") " pod="openshift-network-operator/iptables-alerter-nx756" Sep 10 22:04:59.948209 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947364 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-env-overrides\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.948209 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947380 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-ovnkube-script-lib\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.948209 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947405 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-hostroot\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:04:59.948209 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947420 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02564ad8-b933-4bc4-9e94-8995c093b6b0-cni-binary-copy\") pod \"multus-additional-cni-plugins-lzxzf\" (UID: \"02564ad8-b933-4bc4-9e94-8995c093b6b0\") " pod="openshift-multus/multus-additional-cni-plugins-lzxzf" Sep 10 22:04:59.948209 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947431 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6d6ce980-2a0b-474d-afe7-a72bd2e04f57-socket-dir\") pod \"aws-ebs-csi-driver-node-7lbpg\" (UID: \"6d6ce980-2a0b-474d-afe7-a72bd2e04f57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" Sep 10 22:04:59.948209 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947455 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6d6ce980-2a0b-474d-afe7-a72bd2e04f57-sys-fs\") pod \"aws-ebs-csi-driver-node-7lbpg\" (UID: \"6d6ce980-2a0b-474d-afe7-a72bd2e04f57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" Sep 10 22:04:59.948209 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947498 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5d91688c-bfc4-4fea-90c5-0c45ebfab0b5-host-slash\") pod \"iptables-alerter-nx756\" (UID: \"5d91688c-bfc4-4fea-90c5-0c45ebfab0b5\") " pod="openshift-network-operator/iptables-alerter-nx756" Sep 10 22:04:59.948209 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947535 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-host-cni-netd\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.948209 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947574 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95rqt\" (UniqueName: \"kubernetes.io/projected/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-kube-api-access-95rqt\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.948209 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947584 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-host-cni-netd\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.948209 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947585 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5d91688c-bfc4-4fea-90c5-0c45ebfab0b5-host-slash\") pod \"iptables-alerter-nx756\" (UID: \"5d91688c-bfc4-4fea-90c5-0c45ebfab0b5\") " pod="openshift-network-operator/iptables-alerter-nx756" Sep 10 22:04:59.948209 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947616 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c7114320-6f0b-4cc6-85a8-309ddda645af-serviceca\") pod \"node-ca-ph2dv\" (UID: \"c7114320-6f0b-4cc6-85a8-309ddda645af\") " pod="openshift-image-registry/node-ca-ph2dv" Sep 10 22:04:59.948209 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947657 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tl88\" (UniqueName: \"kubernetes.io/projected/0638522d-7b41-42c0-a93c-61fe7e00935b-kube-api-access-6tl88\") pod \"node-resolver-zgwrg\" (UID: \"0638522d-7b41-42c0-a93c-61fe7e00935b\") " pod="openshift-dns/node-resolver-zgwrg" Sep 10 22:04:59.948209 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947684 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-multus-conf-dir\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:04:59.948209 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947693 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-ovnkube-config\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.948209 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947709 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-etc-sysconfig\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:04:59.948896 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947733 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-etc-sysctl-conf\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:04:59.948896 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947753 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-env-overrides\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.948896 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947756 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-var-lib-kubelet\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:04:59.948896 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947802 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82j6p\" (UniqueName: \"kubernetes.io/projected/d748c035-3205-4b2f-8b7e-a229ff50e466-kube-api-access-82j6p\") pod \"network-metrics-daemon-fp72v\" (UID: \"d748c035-3205-4b2f-8b7e-a229ff50e466\") " pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:04:59.948896 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947849 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5d91688c-bfc4-4fea-90c5-0c45ebfab0b5-iptables-alerter-script\") pod \"iptables-alerter-nx756\" (UID: \"5d91688c-bfc4-4fea-90c5-0c45ebfab0b5\") " pod="openshift-network-operator/iptables-alerter-nx756" Sep 10 22:04:59.948896 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947869 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-run-openvswitch\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.948896 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947895 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-host-var-lib-cni-bin\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:04:59.948896 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947921 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-host-run-netns\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.948896 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947961 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-run-openvswitch\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.948896 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.947976 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-ovn-node-metrics-cert\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.948896 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948021 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-host-run-netns\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.948896 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948021 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-ovnkube-script-lib\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.948896 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948030 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/673499d4-211d-40e2-8a3f-95c74bc1d0f3-original-pull-secret\") pod \"global-pull-secret-syncer-m6nqn\" (UID: \"673499d4-211d-40e2-8a3f-95c74bc1d0f3\") " pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:04:59.948896 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948077 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc7tv\" (UniqueName: \"kubernetes.io/projected/c7114320-6f0b-4cc6-85a8-309ddda645af-kube-api-access-fc7tv\") pod \"node-ca-ph2dv\" (UID: \"c7114320-6f0b-4cc6-85a8-309ddda645af\") " pod="openshift-image-registry/node-ca-ph2dv" Sep 10 22:04:59.948896 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948109 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-host-run-k8s-cni-cncf-io\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:04:59.948896 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948149 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-etc-kubernetes\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:04:59.948896 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948186 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b077b17c-710b-474e-906e-8f86682b86a2-etc-tuned\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:04:59.949555 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948227 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-node-log\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.949555 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948257 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/02564ad8-b933-4bc4-9e94-8995c093b6b0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lzxzf\" (UID: \"02564ad8-b933-4bc4-9e94-8995c093b6b0\") " pod="openshift-multus/multus-additional-cni-plugins-lzxzf" Sep 10 22:04:59.949555 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948281 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctzb2\" (UniqueName: \"kubernetes.io/projected/02564ad8-b933-4bc4-9e94-8995c093b6b0-kube-api-access-ctzb2\") pod \"multus-additional-cni-plugins-lzxzf\" (UID: \"02564ad8-b933-4bc4-9e94-8995c093b6b0\") " pod="openshift-multus/multus-additional-cni-plugins-lzxzf" Sep 10 22:04:59.949555 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948305 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-etc-openvswitch\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.949555 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948324 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-node-log\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.949555 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948330 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-host-run-ovn-kubernetes\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.949555 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948363 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/673499d4-211d-40e2-8a3f-95c74bc1d0f3-dbus\") pod \"global-pull-secret-syncer-m6nqn\" (UID: \"673499d4-211d-40e2-8a3f-95c74bc1d0f3\") " pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:04:59.949555 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948370 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-host-run-ovn-kubernetes\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.949555 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948392 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltfl6\" (UniqueName: \"kubernetes.io/projected/b077b17c-710b-474e-906e-8f86682b86a2-kube-api-access-ltfl6\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:04:59.949555 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948418 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-etc-openvswitch\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.949555 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948424 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0638522d-7b41-42c0-a93c-61fe7e00935b-hosts-file\") pod \"node-resolver-zgwrg\" (UID: \"0638522d-7b41-42c0-a93c-61fe7e00935b\") " pod="openshift-dns/node-resolver-zgwrg" Sep 10 22:04:59.949555 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948472 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-host-kubelet\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.949555 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948512 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-host-kubelet\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.949555 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948523 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-host-slash\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.949555 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948556 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-etc-kubernetes\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:04:59.949555 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948480 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5d91688c-bfc4-4fea-90c5-0c45ebfab0b5-iptables-alerter-script\") pod \"iptables-alerter-nx756\" (UID: \"5d91688c-bfc4-4fea-90c5-0c45ebfab0b5\") " pod="openshift-network-operator/iptables-alerter-nx756" Sep 10 22:04:59.949555 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948477 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0638522d-7b41-42c0-a93c-61fe7e00935b-hosts-file\") pod \"node-resolver-zgwrg\" (UID: \"0638522d-7b41-42c0-a93c-61fe7e00935b\") " pod="openshift-dns/node-resolver-zgwrg" Sep 10 22:04:59.950148 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948558 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-host-slash\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.950148 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948601 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-sys\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:04:59.950148 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948641 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-multus-socket-dir-parent\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:04:59.950148 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948674 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/039f078c-f211-4f4f-8878-d39a9e529672-multus-daemon-config\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:04:59.950148 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948664 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 22:04:59.950148 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948732 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/02564ad8-b933-4bc4-9e94-8995c093b6b0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lzxzf\" (UID: \"02564ad8-b933-4bc4-9e94-8995c093b6b0\") " pod="openshift-multus/multus-additional-cni-plugins-lzxzf" Sep 10 22:04:59.950148 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.948783 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-etc-modprobe-d\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:04:59.952200 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.952182 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-ovn-node-metrics-cert\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:04:59.962497 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.962468 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tl88\" (UniqueName: \"kubernetes.io/projected/0638522d-7b41-42c0-a93c-61fe7e00935b-kube-api-access-6tl88\") pod \"node-resolver-zgwrg\" (UID: \"0638522d-7b41-42c0-a93c-61fe7e00935b\") " pod="openshift-dns/node-resolver-zgwrg" Sep 10 22:04:59.962497 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.962482 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctzb2\" (UniqueName: \"kubernetes.io/projected/02564ad8-b933-4bc4-9e94-8995c093b6b0-kube-api-access-ctzb2\") pod \"multus-additional-cni-plugins-lzxzf\" (UID: \"02564ad8-b933-4bc4-9e94-8995c093b6b0\") " pod="openshift-multus/multus-additional-cni-plugins-lzxzf" Sep 10 22:04:59.962633 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.962478 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82j6p\" (UniqueName: \"kubernetes.io/projected/d748c035-3205-4b2f-8b7e-a229ff50e466-kube-api-access-82j6p\") pod \"network-metrics-daemon-fp72v\" (UID: \"d748c035-3205-4b2f-8b7e-a229ff50e466\") " pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:04:59.962874 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.962859 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tbgb\" (UniqueName: \"kubernetes.io/projected/5d91688c-bfc4-4fea-90c5-0c45ebfab0b5-kube-api-access-8tbgb\") pod \"iptables-alerter-nx756\" (UID: \"5d91688c-bfc4-4fea-90c5-0c45ebfab0b5\") " pod="openshift-network-operator/iptables-alerter-nx756" Sep 10 22:04:59.963102 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:04:59.963085 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95rqt\" (UniqueName: \"kubernetes.io/projected/a7cb2d62-f2e4-4b3e-857e-908d0e1fd121-kube-api-access-95rqt\") pod \"ovnkube-node-5j226\" (UID: \"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121\") " pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:05:00.049642 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.049604 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-hostroot\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.049642 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.049641 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6d6ce980-2a0b-474d-afe7-a72bd2e04f57-socket-dir\") pod \"aws-ebs-csi-driver-node-7lbpg\" (UID: \"6d6ce980-2a0b-474d-afe7-a72bd2e04f57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" Sep 10 22:05:00.049890 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.049660 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6d6ce980-2a0b-474d-afe7-a72bd2e04f57-sys-fs\") pod \"aws-ebs-csi-driver-node-7lbpg\" (UID: \"6d6ce980-2a0b-474d-afe7-a72bd2e04f57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" Sep 10 22:05:00.049890 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.049684 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c7114320-6f0b-4cc6-85a8-309ddda645af-serviceca\") pod \"node-ca-ph2dv\" (UID: \"c7114320-6f0b-4cc6-85a8-309ddda645af\") " pod="openshift-image-registry/node-ca-ph2dv" Sep 10 22:05:00.049890 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.049708 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-multus-conf-dir\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.049890 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.049726 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-etc-sysconfig\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.049890 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.049721 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-hostroot\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.049890 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.049740 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-etc-sysctl-conf\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.049890 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.049758 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-var-lib-kubelet\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.049890 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.049759 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6d6ce980-2a0b-474d-afe7-a72bd2e04f57-sys-fs\") pod \"aws-ebs-csi-driver-node-7lbpg\" (UID: \"6d6ce980-2a0b-474d-afe7-a72bd2e04f57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" Sep 10 22:05:00.049890 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.049783 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-host-var-lib-cni-bin\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.049890 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.049787 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-multus-conf-dir\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.049890 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.049818 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6d6ce980-2a0b-474d-afe7-a72bd2e04f57-socket-dir\") pod \"aws-ebs-csi-driver-node-7lbpg\" (UID: \"6d6ce980-2a0b-474d-afe7-a72bd2e04f57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" Sep 10 22:05:00.049890 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.049824 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-host-var-lib-cni-bin\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.049890 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.049831 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/673499d4-211d-40e2-8a3f-95c74bc1d0f3-original-pull-secret\") pod \"global-pull-secret-syncer-m6nqn\" (UID: \"673499d4-211d-40e2-8a3f-95c74bc1d0f3\") " pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:00.049890 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.049867 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc7tv\" (UniqueName: \"kubernetes.io/projected/c7114320-6f0b-4cc6-85a8-309ddda645af-kube-api-access-fc7tv\") pod \"node-ca-ph2dv\" (UID: \"c7114320-6f0b-4cc6-85a8-309ddda645af\") " pod="openshift-image-registry/node-ca-ph2dv" Sep 10 22:05:00.049890 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.049878 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-var-lib-kubelet\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.049890 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.049877 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-etc-sysconfig\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.049890 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.049890 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-host-run-k8s-cni-cncf-io\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.050822 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:00.049921 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 22:05:00.050822 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.050000 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-etc-sysctl-conf\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.050822 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.050037 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-host-run-k8s-cni-cncf-io\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.050822 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.050069 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-etc-kubernetes\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.050822 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:00.050102 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/673499d4-211d-40e2-8a3f-95c74bc1d0f3-original-pull-secret podName:673499d4-211d-40e2-8a3f-95c74bc1d0f3 nodeName:}" failed. No retries permitted until 2026-09-10 22:05:00.55006899 +0000 UTC m=+2.119558942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/673499d4-211d-40e2-8a3f-95c74bc1d0f3-original-pull-secret") pod "global-pull-secret-syncer-m6nqn" (UID: "673499d4-211d-40e2-8a3f-95c74bc1d0f3") : object "kube-system"/"original-pull-secret" not registered Sep 10 22:05:00.050822 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.050244 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b077b17c-710b-474e-906e-8f86682b86a2-etc-tuned\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.050822 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.050517 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/673499d4-211d-40e2-8a3f-95c74bc1d0f3-dbus\") pod \"global-pull-secret-syncer-m6nqn\" (UID: \"673499d4-211d-40e2-8a3f-95c74bc1d0f3\") " pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:00.050822 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.050543 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltfl6\" (UniqueName: \"kubernetes.io/projected/b077b17c-710b-474e-906e-8f86682b86a2-kube-api-access-ltfl6\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.050822 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.050570 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-etc-kubernetes\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.050822 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.050264 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-etc-kubernetes\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.050822 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.050612 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c7114320-6f0b-4cc6-85a8-309ddda645af-serviceca\") pod \"node-ca-ph2dv\" (UID: \"c7114320-6f0b-4cc6-85a8-309ddda645af\") " pod="openshift-image-registry/node-ca-ph2dv" Sep 10 22:05:00.050822 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.050651 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-etc-kubernetes\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.050822 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.050711 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-sys\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.050822 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.050771 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-sys\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.050822 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.050758 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-multus-socket-dir-parent\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.050822 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.050813 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/039f078c-f211-4f4f-8878-d39a9e529672-multus-daemon-config\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.051659 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.050836 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-etc-modprobe-d\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.051659 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.050924 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/673499d4-211d-40e2-8a3f-95c74bc1d0f3-dbus\") pod \"global-pull-secret-syncer-m6nqn\" (UID: \"673499d4-211d-40e2-8a3f-95c74bc1d0f3\") " pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:00.051659 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.050923 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-multus-socket-dir-parent\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.051659 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051003 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6d6ce980-2a0b-474d-afe7-a72bd2e04f57-device-dir\") pod \"aws-ebs-csi-driver-node-7lbpg\" (UID: \"6d6ce980-2a0b-474d-afe7-a72bd2e04f57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" Sep 10 22:05:00.051659 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051051 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6d6ce980-2a0b-474d-afe7-a72bd2e04f57-etc-selinux\") pod \"aws-ebs-csi-driver-node-7lbpg\" (UID: \"6d6ce980-2a0b-474d-afe7-a72bd2e04f57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" Sep 10 22:05:00.051659 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051076 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4f2170c9-cacb-47f3-8a42-b80039deefef-agent-certs\") pod \"konnectivity-agent-96t46\" (UID: \"4f2170c9-cacb-47f3-8a42-b80039deefef\") " pod="kube-system/konnectivity-agent-96t46" Sep 10 22:05:00.051659 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051084 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-etc-modprobe-d\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.051659 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051090 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6d6ce980-2a0b-474d-afe7-a72bd2e04f57-device-dir\") pod \"aws-ebs-csi-driver-node-7lbpg\" (UID: \"6d6ce980-2a0b-474d-afe7-a72bd2e04f57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" Sep 10 22:05:00.051659 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051124 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7114320-6f0b-4cc6-85a8-309ddda645af-host\") pod \"node-ca-ph2dv\" (UID: \"c7114320-6f0b-4cc6-85a8-309ddda645af\") " pod="openshift-image-registry/node-ca-ph2dv" Sep 10 22:05:00.051659 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051149 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-system-cni-dir\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.051659 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051173 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/039f078c-f211-4f4f-8878-d39a9e529672-cni-binary-copy\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.051659 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051181 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7114320-6f0b-4cc6-85a8-309ddda645af-host\") pod \"node-ca-ph2dv\" (UID: \"c7114320-6f0b-4cc6-85a8-309ddda645af\") " pod="openshift-image-registry/node-ca-ph2dv" Sep 10 22:05:00.051659 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051215 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-host-var-lib-kubelet\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.051659 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051224 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6d6ce980-2a0b-474d-afe7-a72bd2e04f57-etc-selinux\") pod \"aws-ebs-csi-driver-node-7lbpg\" (UID: \"6d6ce980-2a0b-474d-afe7-a72bd2e04f57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" Sep 10 22:05:00.051659 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051240 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6d6ce980-2a0b-474d-afe7-a72bd2e04f57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7lbpg\" (UID: \"6d6ce980-2a0b-474d-afe7-a72bd2e04f57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" Sep 10 22:05:00.051659 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051272 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6d6ce980-2a0b-474d-afe7-a72bd2e04f57-registration-dir\") pod \"aws-ebs-csi-driver-node-7lbpg\" (UID: \"6d6ce980-2a0b-474d-afe7-a72bd2e04f57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" Sep 10 22:05:00.051659 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051306 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-run\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.051659 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051331 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-host\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.052644 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051354 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b077b17c-710b-474e-906e-8f86682b86a2-tmp\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.052644 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051382 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-host-run-multus-certs\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.052644 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051438 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/673499d4-211d-40e2-8a3f-95c74bc1d0f3-kubelet-config\") pod \"global-pull-secret-syncer-m6nqn\" (UID: \"673499d4-211d-40e2-8a3f-95c74bc1d0f3\") " pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:00.052644 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051478 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-etc-sysctl-d\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.052644 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051502 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-etc-systemd\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.052644 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051526 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-lib-modules\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.052644 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051573 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-765sx\" (UniqueName: \"kubernetes.io/projected/493e7732-8863-4f0f-b93e-bce47d8cf3de-kube-api-access-765sx\") pod \"network-check-target-wfb5h\" (UID: \"493e7732-8863-4f0f-b93e-bce47d8cf3de\") " pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:05:00.052644 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051589 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6d6ce980-2a0b-474d-afe7-a72bd2e04f57-registration-dir\") pod \"aws-ebs-csi-driver-node-7lbpg\" (UID: \"6d6ce980-2a0b-474d-afe7-a72bd2e04f57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" Sep 10 22:05:00.052644 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051627 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-host-run-netns\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.052644 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051652 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-run\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.052644 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051664 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/039f078c-f211-4f4f-8878-d39a9e529672-multus-daemon-config\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.052644 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051666 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scp8w\" (UniqueName: \"kubernetes.io/projected/039f078c-f211-4f4f-8878-d39a9e529672-kube-api-access-scp8w\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.052644 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051707 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-host\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.052644 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051710 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-etc-sysctl-d\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.052644 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051738 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/673499d4-211d-40e2-8a3f-95c74bc1d0f3-kubelet-config\") pod \"global-pull-secret-syncer-m6nqn\" (UID: \"673499d4-211d-40e2-8a3f-95c74bc1d0f3\") " pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:00.052644 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051775 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-host-run-multus-certs\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.052644 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051822 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-host-var-lib-kubelet\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.052644 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051836 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-etc-systemd\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.053224 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051869 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-system-cni-dir\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.053224 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051914 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zs5s\" (UniqueName: \"kubernetes.io/projected/6d6ce980-2a0b-474d-afe7-a72bd2e04f57-kube-api-access-4zs5s\") pod \"aws-ebs-csi-driver-node-7lbpg\" (UID: \"6d6ce980-2a0b-474d-afe7-a72bd2e04f57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" Sep 10 22:05:00.053224 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051939 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-host-run-netns\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.053224 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.051965 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6d6ce980-2a0b-474d-afe7-a72bd2e04f57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7lbpg\" (UID: \"6d6ce980-2a0b-474d-afe7-a72bd2e04f57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" Sep 10 22:05:00.053224 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.052001 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4f2170c9-cacb-47f3-8a42-b80039deefef-konnectivity-ca\") pod \"konnectivity-agent-96t46\" (UID: \"4f2170c9-cacb-47f3-8a42-b80039deefef\") " pod="kube-system/konnectivity-agent-96t46" Sep 10 22:05:00.053224 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.052072 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b077b17c-710b-474e-906e-8f86682b86a2-lib-modules\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.053224 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.052233 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-multus-cni-dir\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.053224 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.052240 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/039f078c-f211-4f4f-8878-d39a9e529672-cni-binary-copy\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.053224 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.052279 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-cnibin\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.053224 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.052342 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-multus-cni-dir\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.053224 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.052403 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-host-var-lib-cni-multus\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.053224 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.052427 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-cnibin\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.053224 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.052461 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/039f078c-f211-4f4f-8878-d39a9e529672-host-var-lib-cni-multus\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.053224 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.052586 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4f2170c9-cacb-47f3-8a42-b80039deefef-konnectivity-ca\") pod \"konnectivity-agent-96t46\" (UID: \"4f2170c9-cacb-47f3-8a42-b80039deefef\") " pod="kube-system/konnectivity-agent-96t46" Sep 10 22:05:00.053683 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.053443 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b077b17c-710b-474e-906e-8f86682b86a2-etc-tuned\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.054013 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.053993 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4f2170c9-cacb-47f3-8a42-b80039deefef-agent-certs\") pod \"konnectivity-agent-96t46\" (UID: \"4f2170c9-cacb-47f3-8a42-b80039deefef\") " pod="kube-system/konnectivity-agent-96t46" Sep 10 22:05:00.054294 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.054268 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b077b17c-710b-474e-906e-8f86682b86a2-tmp\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.062355 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:00.062329 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 22:05:00.062355 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:00.062358 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 22:05:00.062598 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:00.062374 2564 projected.go:196] Error preparing data for projected volume kube-api-access-765sx for pod openshift-network-diagnostics/network-check-target-wfb5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 22:05:00.062598 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.062422 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc7tv\" (UniqueName: \"kubernetes.io/projected/c7114320-6f0b-4cc6-85a8-309ddda645af-kube-api-access-fc7tv\") pod \"node-ca-ph2dv\" (UID: \"c7114320-6f0b-4cc6-85a8-309ddda645af\") " pod="openshift-image-registry/node-ca-ph2dv" Sep 10 22:05:00.062598 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:00.062443 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/493e7732-8863-4f0f-b93e-bce47d8cf3de-kube-api-access-765sx podName:493e7732-8863-4f0f-b93e-bce47d8cf3de nodeName:}" failed. No retries permitted until 2026-09-10 22:05:00.5624257 +0000 UTC m=+2.131915677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-765sx" (UniqueName: "kubernetes.io/projected/493e7732-8863-4f0f-b93e-bce47d8cf3de-kube-api-access-765sx") pod "network-check-target-wfb5h" (UID: "493e7732-8863-4f0f-b93e-bce47d8cf3de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 22:05:00.062774 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.062718 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltfl6\" (UniqueName: \"kubernetes.io/projected/b077b17c-710b-474e-906e-8f86682b86a2-kube-api-access-ltfl6\") pod \"tuned-8gs2q\" (UID: \"b077b17c-710b-474e-906e-8f86682b86a2\") " pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.064121 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.064101 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zs5s\" (UniqueName: \"kubernetes.io/projected/6d6ce980-2a0b-474d-afe7-a72bd2e04f57-kube-api-access-4zs5s\") pod \"aws-ebs-csi-driver-node-7lbpg\" (UID: \"6d6ce980-2a0b-474d-afe7-a72bd2e04f57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" Sep 10 22:05:00.064289 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.064273 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scp8w\" (UniqueName: \"kubernetes.io/projected/039f078c-f211-4f4f-8878-d39a9e529672-kube-api-access-scp8w\") pod \"multus-95n7s\" (UID: \"039f078c-f211-4f4f-8878-d39a9e529672\") " pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.176849 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.176767 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nx756" Sep 10 22:05:00.189893 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:05:00.189845 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09bb482ad5b644a06b2ecf966b25fc2.slice/crio-34c71657d59f3ac9221a367a5d7cff7ceba001d714cd0274fd63ec82900cb0b6 WatchSource:0}: Error finding container 34c71657d59f3ac9221a367a5d7cff7ceba001d714cd0274fd63ec82900cb0b6: Status 404 returned error can't find the container with id 34c71657d59f3ac9221a367a5d7cff7ceba001d714cd0274fd63ec82900cb0b6 Sep 10 22:05:00.190036 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.189964 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:05:00.190111 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:05:00.190088 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d91688c_bfc4_4fea_90c5_0c45ebfab0b5.slice/crio-37096bb3f9ff5f204a0ecfbe9e2fd4ea2a1ffb166de16c37c0814a480f5918ad WatchSource:0}: Error finding container 37096bb3f9ff5f204a0ecfbe9e2fd4ea2a1ffb166de16c37c0814a480f5918ad: Status 404 returned error can't find the container with id 37096bb3f9ff5f204a0ecfbe9e2fd4ea2a1ffb166de16c37c0814a480f5918ad Sep 10 22:05:00.198615 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.198593 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 22:05:00.206666 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:05:00.206640 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7cb2d62_f2e4_4b3e_857e_908d0e1fd121.slice/crio-6bce1c57a59c42d370e8e0d10b4dd3d3d8304901bf36de665add50412a991021 WatchSource:0}: Error finding container 6bce1c57a59c42d370e8e0d10b4dd3d3d8304901bf36de665add50412a991021: Status 404 returned error can't find the container with id 6bce1c57a59c42d370e8e0d10b4dd3d3d8304901bf36de665add50412a991021 Sep 10 22:05:00.207152 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:05:00.207068 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d26a6c9bf1e3308cfe50ca1c9f6711.slice/crio-84aa3313c0a9f17344b06bfa28e6c511f3b52ad3e07a13f79fe3dbf1730d1c98 WatchSource:0}: Error finding container 84aa3313c0a9f17344b06bfa28e6c511f3b52ad3e07a13f79fe3dbf1730d1c98: Status 404 returned error can't find the container with id 84aa3313c0a9f17344b06bfa28e6c511f3b52ad3e07a13f79fe3dbf1730d1c98 Sep 10 22:05:00.207326 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.207306 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zgwrg" Sep 10 22:05:00.216504 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.216484 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lzxzf" Sep 10 22:05:00.217832 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:05:00.217808 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0638522d_7b41_42c0_a93c_61fe7e00935b.slice/crio-97c285d70acc06097b0f2d9d984165ffe37739d2d4ddc7d2da61a9922cfa26b3 WatchSource:0}: Error finding container 97c285d70acc06097b0f2d9d984165ffe37739d2d4ddc7d2da61a9922cfa26b3: Status 404 returned error can't find the container with id 97c285d70acc06097b0f2d9d984165ffe37739d2d4ddc7d2da61a9922cfa26b3 Sep 10 22:05:00.222087 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.222068 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 22:05:00.226434 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:05:00.226412 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02564ad8_b933_4bc4_9e94_8995c093b6b0.slice/crio-425956d1fdb6437cdcf9f70cff3ed079b39530e83cf699f4d559c3daea532b25 WatchSource:0}: Error finding container 425956d1fdb6437cdcf9f70cff3ed079b39530e83cf699f4d559c3daea532b25: Status 404 returned error can't find the container with id 425956d1fdb6437cdcf9f70cff3ed079b39530e83cf699f4d559c3daea532b25 Sep 10 22:05:00.235049 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.235022 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ph2dv" Sep 10 22:05:00.241082 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.241060 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-95n7s" Sep 10 22:05:00.243668 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:05:00.243643 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7114320_6f0b_4cc6_85a8_309ddda645af.slice/crio-6087623da50b26a67c2e7e529d577e8a540e899ae6db9d28a7837cb2081b081f WatchSource:0}: Error finding container 6087623da50b26a67c2e7e529d577e8a540e899ae6db9d28a7837cb2081b081f: Status 404 returned error can't find the container with id 6087623da50b26a67c2e7e529d577e8a540e899ae6db9d28a7837cb2081b081f Sep 10 22:05:00.249300 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:05:00.249279 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod039f078c_f211_4f4f_8878_d39a9e529672.slice/crio-04ff9c81dc535866a5b6fb36528b1fc37ad4aebc7c643b293e3f3feac59a56fc WatchSource:0}: Error finding container 04ff9c81dc535866a5b6fb36528b1fc37ad4aebc7c643b293e3f3feac59a56fc: Status 404 returned error can't find the container with id 04ff9c81dc535866a5b6fb36528b1fc37ad4aebc7c643b293e3f3feac59a56fc Sep 10 22:05:00.277500 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.277459 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" Sep 10 22:05:00.287217 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:05:00.287178 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb077b17c_710b_474e_906e_8f86682b86a2.slice/crio-2e4def4bb71ebb88097f97771be68ad49b08d95d45a2ff9ffaa8e4835c770762 WatchSource:0}: Error finding container 2e4def4bb71ebb88097f97771be68ad49b08d95d45a2ff9ffaa8e4835c770762: Status 404 returned error can't find the container with id 2e4def4bb71ebb88097f97771be68ad49b08d95d45a2ff9ffaa8e4835c770762 Sep 10 22:05:00.291439 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.291418 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" Sep 10 22:05:00.296455 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.296437 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-96t46" Sep 10 22:05:00.300398 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:05:00.300377 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6ce980_2a0b_474d_afe7_a72bd2e04f57.slice/crio-e56403a07739f0a416d2f72707572986abc17183e40d1547f240c1706cb9428b WatchSource:0}: Error finding container e56403a07739f0a416d2f72707572986abc17183e40d1547f240c1706cb9428b: Status 404 returned error can't find the container with id e56403a07739f0a416d2f72707572986abc17183e40d1547f240c1706cb9428b Sep 10 22:05:00.308415 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:00.308385 2564 kuberuntime_manager.go:1459] "Unhandled Error" err="container konnectivity-agent start failed in pod konnectivity-agent-96t46_kube-system(4f2170c9-cacb-47f3-8a42-b80039deefef): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Sep 10 22:05:00.308505 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:00.308430 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"konnectivity-agent\" with ErrImagePull: \"pull QPS exceeded\"" pod="kube-system/konnectivity-agent-96t46" podUID="4f2170c9-cacb-47f3-8a42-b80039deefef" Sep 10 22:05:00.454080 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.453990 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs\") pod \"network-metrics-daemon-fp72v\" (UID: \"d748c035-3205-4b2f-8b7e-a229ff50e466\") " pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:05:00.454236 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:00.454145 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 22:05:00.454236 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:00.454208 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs podName:d748c035-3205-4b2f-8b7e-a229ff50e466 nodeName:}" failed. No retries permitted until 2026-09-10 22:05:01.454191688 +0000 UTC m=+3.023681654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs") pod "network-metrics-daemon-fp72v" (UID: "d748c035-3205-4b2f-8b7e-a229ff50e466") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 22:05:00.554652 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.554619 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/673499d4-211d-40e2-8a3f-95c74bc1d0f3-original-pull-secret\") pod \"global-pull-secret-syncer-m6nqn\" (UID: \"673499d4-211d-40e2-8a3f-95c74bc1d0f3\") " pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:00.554819 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:00.554740 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 22:05:00.554819 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:00.554800 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/673499d4-211d-40e2-8a3f-95c74bc1d0f3-original-pull-secret podName:673499d4-211d-40e2-8a3f-95c74bc1d0f3 nodeName:}" failed. No retries permitted until 2026-09-10 22:05:01.554786208 +0000 UTC m=+3.124276152 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/673499d4-211d-40e2-8a3f-95c74bc1d0f3-original-pull-secret") pod "global-pull-secret-syncer-m6nqn" (UID: "673499d4-211d-40e2-8a3f-95c74bc1d0f3") : object "kube-system"/"original-pull-secret" not registered Sep 10 22:05:00.655449 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.655405 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-765sx\" (UniqueName: \"kubernetes.io/projected/493e7732-8863-4f0f-b93e-bce47d8cf3de-kube-api-access-765sx\") pod \"network-check-target-wfb5h\" (UID: \"493e7732-8863-4f0f-b93e-bce47d8cf3de\") " pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:05:00.655612 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:00.655547 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 22:05:00.655612 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:00.655563 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 22:05:00.655612 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:00.655573 2564 projected.go:196] Error preparing data for projected volume kube-api-access-765sx for pod openshift-network-diagnostics/network-check-target-wfb5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 22:05:00.655717 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:00.655626 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/493e7732-8863-4f0f-b93e-bce47d8cf3de-kube-api-access-765sx podName:493e7732-8863-4f0f-b93e-bce47d8cf3de nodeName:}" failed. No retries permitted until 2026-09-10 22:05:01.655612819 +0000 UTC m=+3.225102780 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-765sx" (UniqueName: "kubernetes.io/projected/493e7732-8863-4f0f-b93e-bce47d8cf3de-kube-api-access-765sx") pod "network-check-target-wfb5h" (UID: "493e7732-8863-4f0f-b93e-bce47d8cf3de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 22:05:00.874091 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.873813 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 22:05:00.896515 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.896407 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 22:00:00 +0000 UTC" deadline="2028-05-21 19:47:27.630906616 +0000 UTC" Sep 10 22:05:00.896515 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.896448 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14853h42m26.734464525s" Sep 10 22:05:00.940315 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.939154 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:05:00.940315 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:00.939430 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp72v" podUID="d748c035-3205-4b2f-8b7e-a229ff50e466" Sep 10 22:05:00.940315 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.940124 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:05:00.940315 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:00.940246 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wfb5h" podUID="493e7732-8863-4f0f-b93e-bce47d8cf3de" Sep 10 22:05:00.966092 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.965721 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5j226" event={"ID":"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121","Type":"ContainerStarted","Data":"6bce1c57a59c42d370e8e0d10b4dd3d3d8304901bf36de665add50412a991021"} Sep 10 22:05:00.978176 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.978131 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nx756" event={"ID":"5d91688c-bfc4-4fea-90c5-0c45ebfab0b5","Type":"ContainerStarted","Data":"37096bb3f9ff5f204a0ecfbe9e2fd4ea2a1ffb166de16c37c0814a480f5918ad"} Sep 10 22:05:00.990300 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:00.990218 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.ec2.internal" event={"ID":"f09bb482ad5b644a06b2ecf966b25fc2","Type":"ContainerStarted","Data":"34c71657d59f3ac9221a367a5d7cff7ceba001d714cd0274fd63ec82900cb0b6"} Sep 10 22:05:01.005149 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:01.005101 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-96t46" event={"ID":"4f2170c9-cacb-47f3-8a42-b80039deefef","Type":"ContainerStarted","Data":"c538ed47fce4d02e6dbc2f37e928cf05fae2844f2c95df2d36f84859077e6a4b"} Sep 10 22:05:01.008208 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:01.007931 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"konnectivity-agent\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb18802deb495c8d9b0ab2c3d8c3043b73eafbad562b04b71a3717870ccdff74\\\": ErrImagePull: pull QPS exceeded\"" pod="kube-system/konnectivity-agent-96t46" podUID="4f2170c9-cacb-47f3-8a42-b80039deefef" Sep 10 22:05:01.009462 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:01.009354 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" event={"ID":"6d6ce980-2a0b-474d-afe7-a72bd2e04f57","Type":"ContainerStarted","Data":"e56403a07739f0a416d2f72707572986abc17183e40d1547f240c1706cb9428b"} Sep 10 22:05:01.015627 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:01.015131 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" event={"ID":"b077b17c-710b-474e-906e-8f86682b86a2","Type":"ContainerStarted","Data":"2e4def4bb71ebb88097f97771be68ad49b08d95d45a2ff9ffaa8e4835c770762"} Sep 10 22:05:01.016940 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:01.016848 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-95n7s" event={"ID":"039f078c-f211-4f4f-8878-d39a9e529672","Type":"ContainerStarted","Data":"04ff9c81dc535866a5b6fb36528b1fc37ad4aebc7c643b293e3f3feac59a56fc"} Sep 10 22:05:01.026432 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:01.026396 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ph2dv" event={"ID":"c7114320-6f0b-4cc6-85a8-309ddda645af","Type":"ContainerStarted","Data":"6087623da50b26a67c2e7e529d577e8a540e899ae6db9d28a7837cb2081b081f"} Sep 10 22:05:01.031018 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:01.030992 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzxzf" event={"ID":"02564ad8-b933-4bc4-9e94-8995c093b6b0","Type":"ContainerStarted","Data":"425956d1fdb6437cdcf9f70cff3ed079b39530e83cf699f4d559c3daea532b25"} Sep 10 22:05:01.033002 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:01.032960 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zgwrg" event={"ID":"0638522d-7b41-42c0-a93c-61fe7e00935b","Type":"ContainerStarted","Data":"97c285d70acc06097b0f2d9d984165ffe37739d2d4ddc7d2da61a9922cfa26b3"} Sep 10 22:05:01.038766 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:01.038733 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" event={"ID":"45d26a6c9bf1e3308cfe50ca1c9f6711","Type":"ContainerStarted","Data":"84aa3313c0a9f17344b06bfa28e6c511f3b52ad3e07a13f79fe3dbf1730d1c98"} Sep 10 22:05:01.463836 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:01.463220 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs\") pod \"network-metrics-daemon-fp72v\" (UID: \"d748c035-3205-4b2f-8b7e-a229ff50e466\") " pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:05:01.463836 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:01.463396 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 22:05:01.463836 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:01.463467 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs podName:d748c035-3205-4b2f-8b7e-a229ff50e466 nodeName:}" failed. No retries permitted until 2026-09-10 22:05:03.463447006 +0000 UTC m=+5.032936956 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs") pod "network-metrics-daemon-fp72v" (UID: "d748c035-3205-4b2f-8b7e-a229ff50e466") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 22:05:01.564039 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:01.563998 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/673499d4-211d-40e2-8a3f-95c74bc1d0f3-original-pull-secret\") pod \"global-pull-secret-syncer-m6nqn\" (UID: \"673499d4-211d-40e2-8a3f-95c74bc1d0f3\") " pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:01.564246 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:01.564156 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 22:05:01.564246 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:01.564223 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/673499d4-211d-40e2-8a3f-95c74bc1d0f3-original-pull-secret podName:673499d4-211d-40e2-8a3f-95c74bc1d0f3 nodeName:}" failed. No retries permitted until 2026-09-10 22:05:03.56420408 +0000 UTC m=+5.133694031 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/673499d4-211d-40e2-8a3f-95c74bc1d0f3-original-pull-secret") pod "global-pull-secret-syncer-m6nqn" (UID: "673499d4-211d-40e2-8a3f-95c74bc1d0f3") : object "kube-system"/"original-pull-secret" not registered Sep 10 22:05:01.665648 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:01.664979 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-765sx\" (UniqueName: \"kubernetes.io/projected/493e7732-8863-4f0f-b93e-bce47d8cf3de-kube-api-access-765sx\") pod \"network-check-target-wfb5h\" (UID: \"493e7732-8863-4f0f-b93e-bce47d8cf3de\") " pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:05:01.665648 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:01.665175 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 22:05:01.665648 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:01.665199 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 22:05:01.665648 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:01.665213 2564 projected.go:196] Error preparing data for projected volume kube-api-access-765sx for pod openshift-network-diagnostics/network-check-target-wfb5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 22:05:01.665648 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:01.665275 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/493e7732-8863-4f0f-b93e-bce47d8cf3de-kube-api-access-765sx podName:493e7732-8863-4f0f-b93e-bce47d8cf3de nodeName:}" failed. No retries permitted until 2026-09-10 22:05:03.665257037 +0000 UTC m=+5.234746985 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-765sx" (UniqueName: "kubernetes.io/projected/493e7732-8863-4f0f-b93e-bce47d8cf3de-kube-api-access-765sx") pod "network-check-target-wfb5h" (UID: "493e7732-8863-4f0f-b93e-bce47d8cf3de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 22:05:01.897741 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:01.897618 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 22:00:00 +0000 UTC" deadline="2028-04-10 15:43:37.664515499 +0000 UTC" Sep 10 22:05:01.897741 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:01.897685 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13865h38m35.766835217s" Sep 10 22:05:01.931431 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:01.931396 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:01.931638 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:01.931548 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6nqn" podUID="673499d4-211d-40e2-8a3f-95c74bc1d0f3" Sep 10 22:05:01.980437 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:01.980404 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 22:05:02.934392 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:02.934351 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:05:02.934902 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:02.934517 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp72v" podUID="d748c035-3205-4b2f-8b7e-a229ff50e466" Sep 10 22:05:02.934994 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:02.934908 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:05:02.935049 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:02.935029 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wfb5h" podUID="493e7732-8863-4f0f-b93e-bce47d8cf3de" Sep 10 22:05:03.482248 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:03.481381 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs\") pod \"network-metrics-daemon-fp72v\" (UID: \"d748c035-3205-4b2f-8b7e-a229ff50e466\") " pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:05:03.482248 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:03.481602 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 22:05:03.482248 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:03.481665 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs podName:d748c035-3205-4b2f-8b7e-a229ff50e466 nodeName:}" failed. No retries permitted until 2026-09-10 22:05:07.481645808 +0000 UTC m=+9.051135766 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs") pod "network-metrics-daemon-fp72v" (UID: "d748c035-3205-4b2f-8b7e-a229ff50e466") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 22:05:03.583487 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:03.582792 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/673499d4-211d-40e2-8a3f-95c74bc1d0f3-original-pull-secret\") pod \"global-pull-secret-syncer-m6nqn\" (UID: \"673499d4-211d-40e2-8a3f-95c74bc1d0f3\") " pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:03.583487 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:03.583013 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 22:05:03.583487 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:03.583081 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/673499d4-211d-40e2-8a3f-95c74bc1d0f3-original-pull-secret podName:673499d4-211d-40e2-8a3f-95c74bc1d0f3 nodeName:}" failed. No retries permitted until 2026-09-10 22:05:07.583062179 +0000 UTC m=+9.152552127 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/673499d4-211d-40e2-8a3f-95c74bc1d0f3-original-pull-secret") pod "global-pull-secret-syncer-m6nqn" (UID: "673499d4-211d-40e2-8a3f-95c74bc1d0f3") : object "kube-system"/"original-pull-secret" not registered Sep 10 22:05:03.686531 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:03.685860 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-765sx\" (UniqueName: \"kubernetes.io/projected/493e7732-8863-4f0f-b93e-bce47d8cf3de-kube-api-access-765sx\") pod \"network-check-target-wfb5h\" (UID: \"493e7732-8863-4f0f-b93e-bce47d8cf3de\") " pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:05:03.686531 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:03.686061 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 22:05:03.686531 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:03.686085 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 22:05:03.686531 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:03.686099 2564 projected.go:196] Error preparing data for projected volume kube-api-access-765sx for pod openshift-network-diagnostics/network-check-target-wfb5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 22:05:03.686531 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:03.686165 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/493e7732-8863-4f0f-b93e-bce47d8cf3de-kube-api-access-765sx podName:493e7732-8863-4f0f-b93e-bce47d8cf3de nodeName:}" failed. No retries permitted until 2026-09-10 22:05:07.686145306 +0000 UTC m=+9.255635275 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-765sx" (UniqueName: "kubernetes.io/projected/493e7732-8863-4f0f-b93e-bce47d8cf3de-kube-api-access-765sx") pod "network-check-target-wfb5h" (UID: "493e7732-8863-4f0f-b93e-bce47d8cf3de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 22:05:03.931175 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:03.931079 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:03.931349 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:03.931242 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6nqn" podUID="673499d4-211d-40e2-8a3f-95c74bc1d0f3" Sep 10 22:05:04.932481 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:04.932378 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:05:04.933135 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:04.932606 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wfb5h" podUID="493e7732-8863-4f0f-b93e-bce47d8cf3de" Sep 10 22:05:04.933469 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:04.932336 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:05:04.933614 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:04.933569 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp72v" podUID="d748c035-3205-4b2f-8b7e-a229ff50e466" Sep 10 22:05:05.931316 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:05.931280 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:05.931502 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:05.931445 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6nqn" podUID="673499d4-211d-40e2-8a3f-95c74bc1d0f3" Sep 10 22:05:06.931707 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:06.931670 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:05:06.932208 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:06.931813 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wfb5h" podUID="493e7732-8863-4f0f-b93e-bce47d8cf3de" Sep 10 22:05:06.932334 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:06.932314 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:05:06.932453 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:06.932430 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp72v" podUID="d748c035-3205-4b2f-8b7e-a229ff50e466" Sep 10 22:05:07.520067 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:07.520025 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs\") pod \"network-metrics-daemon-fp72v\" (UID: \"d748c035-3205-4b2f-8b7e-a229ff50e466\") " pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:05:07.520262 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:07.520217 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 22:05:07.520330 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:07.520305 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs podName:d748c035-3205-4b2f-8b7e-a229ff50e466 nodeName:}" failed. No retries permitted until 2026-09-10 22:05:15.520280739 +0000 UTC m=+17.089770686 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs") pod "network-metrics-daemon-fp72v" (UID: "d748c035-3205-4b2f-8b7e-a229ff50e466") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 22:05:07.621348 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:07.621258 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/673499d4-211d-40e2-8a3f-95c74bc1d0f3-original-pull-secret\") pod \"global-pull-secret-syncer-m6nqn\" (UID: \"673499d4-211d-40e2-8a3f-95c74bc1d0f3\") " pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:07.621540 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:07.621401 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 22:05:07.621540 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:07.621467 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/673499d4-211d-40e2-8a3f-95c74bc1d0f3-original-pull-secret podName:673499d4-211d-40e2-8a3f-95c74bc1d0f3 nodeName:}" failed. No retries permitted until 2026-09-10 22:05:15.621452372 +0000 UTC m=+17.190942316 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/673499d4-211d-40e2-8a3f-95c74bc1d0f3-original-pull-secret") pod "global-pull-secret-syncer-m6nqn" (UID: "673499d4-211d-40e2-8a3f-95c74bc1d0f3") : object "kube-system"/"original-pull-secret" not registered Sep 10 22:05:07.722336 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:07.722291 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-765sx\" (UniqueName: \"kubernetes.io/projected/493e7732-8863-4f0f-b93e-bce47d8cf3de-kube-api-access-765sx\") pod \"network-check-target-wfb5h\" (UID: \"493e7732-8863-4f0f-b93e-bce47d8cf3de\") " pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:05:07.722526 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:07.722498 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 22:05:07.722611 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:07.722528 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 22:05:07.722611 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:07.722543 2564 projected.go:196] Error preparing data for projected volume kube-api-access-765sx for pod openshift-network-diagnostics/network-check-target-wfb5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 22:05:07.722724 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:07.722650 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/493e7732-8863-4f0f-b93e-bce47d8cf3de-kube-api-access-765sx podName:493e7732-8863-4f0f-b93e-bce47d8cf3de nodeName:}" failed. No retries permitted until 2026-09-10 22:05:15.7226294 +0000 UTC m=+17.292119350 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-765sx" (UniqueName: "kubernetes.io/projected/493e7732-8863-4f0f-b93e-bce47d8cf3de-kube-api-access-765sx") pod "network-check-target-wfb5h" (UID: "493e7732-8863-4f0f-b93e-bce47d8cf3de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 22:05:07.931463 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:07.931301 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:07.931463 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:07.931429 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6nqn" podUID="673499d4-211d-40e2-8a3f-95c74bc1d0f3" Sep 10 22:05:08.932769 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:08.932490 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:05:08.932769 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:08.932604 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wfb5h" podUID="493e7732-8863-4f0f-b93e-bce47d8cf3de" Sep 10 22:05:08.932769 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:08.932618 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:05:08.932769 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:08.932714 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp72v" podUID="d748c035-3205-4b2f-8b7e-a229ff50e466" Sep 10 22:05:09.930885 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:09.930851 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:09.931090 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:09.930992 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6nqn" podUID="673499d4-211d-40e2-8a3f-95c74bc1d0f3" Sep 10 22:05:10.931634 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:10.931599 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:05:10.932152 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:10.931734 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wfb5h" podUID="493e7732-8863-4f0f-b93e-bce47d8cf3de" Sep 10 22:05:10.932152 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:10.931778 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:05:10.932152 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:10.931912 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp72v" podUID="d748c035-3205-4b2f-8b7e-a229ff50e466" Sep 10 22:05:11.931537 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:11.931495 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:11.931708 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:11.931642 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6nqn" podUID="673499d4-211d-40e2-8a3f-95c74bc1d0f3" Sep 10 22:05:12.931412 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:12.931375 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:05:12.931412 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:12.931395 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:05:12.931640 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:12.931508 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wfb5h" podUID="493e7732-8863-4f0f-b93e-bce47d8cf3de" Sep 10 22:05:12.931691 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:12.931648 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp72v" podUID="d748c035-3205-4b2f-8b7e-a229ff50e466" Sep 10 22:05:13.930916 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:13.930873 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:13.931376 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:13.931009 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6nqn" podUID="673499d4-211d-40e2-8a3f-95c74bc1d0f3" Sep 10 22:05:14.931077 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:14.931003 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:05:14.931447 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:14.931012 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:05:14.931447 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:14.931111 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wfb5h" podUID="493e7732-8863-4f0f-b93e-bce47d8cf3de" Sep 10 22:05:14.931447 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:14.931192 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp72v" podUID="d748c035-3205-4b2f-8b7e-a229ff50e466" Sep 10 22:05:15.578026 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:15.577974 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs\") pod \"network-metrics-daemon-fp72v\" (UID: \"d748c035-3205-4b2f-8b7e-a229ff50e466\") " pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:05:15.578198 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:15.578124 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 22:05:15.578253 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:15.578201 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs podName:d748c035-3205-4b2f-8b7e-a229ff50e466 nodeName:}" failed. No retries permitted until 2026-09-10 22:05:31.578181961 +0000 UTC m=+33.147671906 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs") pod "network-metrics-daemon-fp72v" (UID: "d748c035-3205-4b2f-8b7e-a229ff50e466") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 22:05:15.678759 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:15.678714 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/673499d4-211d-40e2-8a3f-95c74bc1d0f3-original-pull-secret\") pod \"global-pull-secret-syncer-m6nqn\" (UID: \"673499d4-211d-40e2-8a3f-95c74bc1d0f3\") " pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:15.678964 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:15.678873 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 22:05:15.679047 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:15.678966 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/673499d4-211d-40e2-8a3f-95c74bc1d0f3-original-pull-secret podName:673499d4-211d-40e2-8a3f-95c74bc1d0f3 nodeName:}" failed. No retries permitted until 2026-09-10 22:05:31.678946675 +0000 UTC m=+33.248436633 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/673499d4-211d-40e2-8a3f-95c74bc1d0f3-original-pull-secret") pod "global-pull-secret-syncer-m6nqn" (UID: "673499d4-211d-40e2-8a3f-95c74bc1d0f3") : object "kube-system"/"original-pull-secret" not registered Sep 10 22:05:15.779846 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:15.779804 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-765sx\" (UniqueName: \"kubernetes.io/projected/493e7732-8863-4f0f-b93e-bce47d8cf3de-kube-api-access-765sx\") pod \"network-check-target-wfb5h\" (UID: \"493e7732-8863-4f0f-b93e-bce47d8cf3de\") " pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:05:15.780048 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:15.779981 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 22:05:15.780048 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:15.779996 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 22:05:15.780048 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:15.780006 2564 projected.go:196] Error preparing data for projected volume kube-api-access-765sx for pod openshift-network-diagnostics/network-check-target-wfb5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 22:05:15.780199 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:15.780061 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/493e7732-8863-4f0f-b93e-bce47d8cf3de-kube-api-access-765sx podName:493e7732-8863-4f0f-b93e-bce47d8cf3de nodeName:}" failed. No retries permitted until 2026-09-10 22:05:31.780047722 +0000 UTC m=+33.349537665 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-765sx" (UniqueName: "kubernetes.io/projected/493e7732-8863-4f0f-b93e-bce47d8cf3de-kube-api-access-765sx") pod "network-check-target-wfb5h" (UID: "493e7732-8863-4f0f-b93e-bce47d8cf3de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 22:05:15.931397 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:15.931363 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:15.931842 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:15.931496 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6nqn" podUID="673499d4-211d-40e2-8a3f-95c74bc1d0f3" Sep 10 22:05:16.933995 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:16.933964 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:05:16.934595 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:16.933965 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:05:16.934595 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:16.934098 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp72v" podUID="d748c035-3205-4b2f-8b7e-a229ff50e466" Sep 10 22:05:16.934595 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:16.934183 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wfb5h" podUID="493e7732-8863-4f0f-b93e-bce47d8cf3de" Sep 10 22:05:17.930933 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:17.930888 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:17.931097 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:17.931032 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6nqn" podUID="673499d4-211d-40e2-8a3f-95c74bc1d0f3" Sep 10 22:05:18.934584 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:18.934478 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:05:18.934584 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:18.934565 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp72v" podUID="d748c035-3205-4b2f-8b7e-a229ff50e466" Sep 10 22:05:18.935071 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:18.934603 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:05:18.935071 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:18.934647 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wfb5h" podUID="493e7732-8863-4f0f-b93e-bce47d8cf3de" Sep 10 22:05:19.100259 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:19.100209 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.ec2.internal" event={"ID":"f09bb482ad5b644a06b2ecf966b25fc2","Type":"ContainerStarted","Data":"a8ee2023080ebca88c7cbe883fc9a01cb7df510afb851be75a43d122731d7967"} Sep 10 22:05:19.102049 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:19.102018 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" event={"ID":"b077b17c-710b-474e-906e-8f86682b86a2","Type":"ContainerStarted","Data":"80b99554cf68a27c89afe69ee69d89df376f6b98d6afb8df06d6cf23fc02c22e"} Sep 10 22:05:19.103803 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:19.103772 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-95n7s" event={"ID":"039f078c-f211-4f4f-8878-d39a9e529672","Type":"ContainerStarted","Data":"94f934fe1d62ba8dfbcedfaa01205709c66f54b52dc3a934357af7b853443aa2"} Sep 10 22:05:19.118589 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:19.118540 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.ec2.internal" podStartSLOduration=19.118527454 podStartE2EDuration="19.118527454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:05:19.118141653 +0000 UTC m=+20.687631619" watchObservedRunningTime="2026-09-10 22:05:19.118527454 +0000 UTC m=+20.688017417" Sep 10 22:05:19.134487 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:19.134441 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8gs2q" podStartSLOduration=1.5926881320000001 podStartE2EDuration="20.134424891s" totalImagesPullingTime="18.541736759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:04:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:05:19.134156806 +0000 UTC m=+20.703646770" watchObservedRunningTime="2026-09-10 22:05:19.134424891 +0000 UTC m=+20.703914854" Sep 10 22:05:19.150949 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:19.150870 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-95n7s" podStartSLOduration=1.530593759 podStartE2EDuration="20.150854221s" totalImagesPullingTime="18.620260462s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:04:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:05:19.150557952 +0000 UTC m=+20.720047916" watchObservedRunningTime="2026-09-10 22:05:19.150854221 +0000 UTC m=+20.720344187" Sep 10 22:05:19.931789 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:19.931624 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:19.931915 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:19.931866 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6nqn" podUID="673499d4-211d-40e2-8a3f-95c74bc1d0f3" Sep 10 22:05:20.107617 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:20.107582 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" event={"ID":"6d6ce980-2a0b-474d-afe7-a72bd2e04f57","Type":"ContainerStarted","Data":"a8c57f32ba4df7994d6ac4b291bb4b82cf0d84df60a313d6aa66413cdc42fca9"} Sep 10 22:05:20.109052 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:20.109029 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ph2dv" event={"ID":"c7114320-6f0b-4cc6-85a8-309ddda645af","Type":"ContainerStarted","Data":"e55566c01cac77d2175c278fb8a806ba409cfe8beb74f06ebc9253bfb9360f79"} Sep 10 22:05:20.110583 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:20.110560 2564 generic.go:356] "Generic (PLEG): container finished" podID="02564ad8-b933-4bc4-9e94-8995c093b6b0" containerID="3de49efd713f6b2f1117f6940b5816eb97c297782e61af63236947fb4191fbab" exitCode=0 Sep 10 22:05:20.110666 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:20.110634 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzxzf" event={"ID":"02564ad8-b933-4bc4-9e94-8995c093b6b0","Type":"ContainerDied","Data":"3de49efd713f6b2f1117f6940b5816eb97c297782e61af63236947fb4191fbab"} Sep 10 22:05:20.111945 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:20.111924 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zgwrg" event={"ID":"0638522d-7b41-42c0-a93c-61fe7e00935b","Type":"ContainerStarted","Data":"fa1a2235e65582aa497dd2d4b800cb27c20169807940ec3c850d507956f4b9bf"} Sep 10 22:05:20.113364 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:20.113344 2564 generic.go:356] "Generic (PLEG): container finished" podID="45d26a6c9bf1e3308cfe50ca1c9f6711" containerID="d7f85a044cff0c83d1ca507fc36e77e738669530d34477ca46e13be497e089f9" exitCode=0 Sep 10 22:05:20.113437 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:20.113373 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" event={"ID":"45d26a6c9bf1e3308cfe50ca1c9f6711","Type":"ContainerDied","Data":"d7f85a044cff0c83d1ca507fc36e77e738669530d34477ca46e13be497e089f9"} Sep 10 22:05:20.115398 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:20.115379 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5j226" event={"ID":"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121","Type":"ContainerStarted","Data":"5c123eeacaddd52b76d073679103a9ab691aee032d09ef6f42469f0d3b19c75a"} Sep 10 22:05:20.115496 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:20.115443 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5j226" event={"ID":"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121","Type":"ContainerStarted","Data":"de3ae3256241340555cbd899cf6f62de2bbb54d4a073ae2d24ce87eaa110b9fa"} Sep 10 22:05:20.115496 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:20.115480 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5j226" event={"ID":"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121","Type":"ContainerStarted","Data":"5e212c85d2a50a83bd7345b5fe582a5a46234e692bc6ecd0b4fc103956300310"} Sep 10 22:05:20.115606 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:20.115509 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5j226" event={"ID":"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121","Type":"ContainerStarted","Data":"413c2237ac915570cdd36e97a919b33fadf83d3ae9dde4f29ad82be693efeeb1"} Sep 10 22:05:20.122675 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:20.122638 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ph2dv" podStartSLOduration=2.558782481 podStartE2EDuration="21.12262585s" totalImagesPullingTime="18.563843369s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:04:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:05:20.122624607 +0000 UTC m=+21.692114571" watchObservedRunningTime="2026-09-10 22:05:20.12262585 +0000 UTC m=+21.692115796" Sep 10 22:05:20.177462 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:20.177359 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zgwrg" podStartSLOduration=2.586868882 podStartE2EDuration="21.177337026s" totalImagesPullingTime="18.590468144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:04:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:05:20.177146549 +0000 UTC m=+21.746636511" watchObservedRunningTime="2026-09-10 22:05:20.177337026 +0000 UTC m=+21.746826989" Sep 10 22:05:20.905226 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:20.904997 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 22:05:20.916499 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:20.916396 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T22:05:20.905206331Z","UUID":"373830ae-886c-4f6b-b55c-570755bb687e","Handler":null,"Name":"","Endpoint":""} Sep 10 22:05:20.918134 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:20.918108 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 22:05:20.918252 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:20.918142 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 22:05:20.935031 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:20.934999 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:05:20.935198 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:20.935000 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:05:20.935198 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:20.935131 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wfb5h" podUID="493e7732-8863-4f0f-b93e-bce47d8cf3de" Sep 10 22:05:20.935286 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:20.935225 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp72v" podUID="d748c035-3205-4b2f-8b7e-a229ff50e466" Sep 10 22:05:21.120529 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:21.120495 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" event={"ID":"6d6ce980-2a0b-474d-afe7-a72bd2e04f57","Type":"ContainerStarted","Data":"6ef3f66c087dfce96c3c87f21a778c2eaccedfd9d1df64201a8f93cc96f343b9"} Sep 10 22:05:21.122926 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:21.122881 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" event={"ID":"45d26a6c9bf1e3308cfe50ca1c9f6711","Type":"ContainerStarted","Data":"d3ed10804e53874ecd960b7783e7b3199b580e5e446fbb83454142d681f2711a"} Sep 10 22:05:21.126264 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:21.126234 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5j226" event={"ID":"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121","Type":"ContainerStarted","Data":"98d6969172eeeac3351a51f1a1d52edbf16f437c181a2bdb8d2fae054a2e0ba3"} Sep 10 22:05:21.126393 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:21.126301 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5j226" event={"ID":"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121","Type":"ContainerStarted","Data":"cb213554b5894a3afa615c864dbd57f1141bbb4a933e11227e12831b21f05b15"} Sep 10 22:05:21.127915 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:21.127876 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nx756" event={"ID":"5d91688c-bfc4-4fea-90c5-0c45ebfab0b5","Type":"ContainerStarted","Data":"13917e273016f101ace491f7ef99adfaf28fd14cff3fac53796c8eda43b4821e"} Sep 10 22:05:21.129436 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:21.129411 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-96t46" event={"ID":"4f2170c9-cacb-47f3-8a42-b80039deefef","Type":"ContainerStarted","Data":"b8bae0c0be14d870759793dcd4e1a0408c437cea11541179610f764289665dc3"} Sep 10 22:05:21.138534 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:21.138485 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.ec2.internal" podStartSLOduration=21.138465023 podStartE2EDuration="21.138465023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:05:21.13793158 +0000 UTC m=+22.707421544" watchObservedRunningTime="2026-09-10 22:05:21.138465023 +0000 UTC m=+22.707954989" Sep 10 22:05:21.153281 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:21.153234 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nx756" podStartSLOduration=3.518200471 podStartE2EDuration="22.153218183s" totalImagesPullingTime="18.635017712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:04:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:05:21.152710489 +0000 UTC m=+22.722200453" watchObservedRunningTime="2026-09-10 22:05:21.153218183 +0000 UTC m=+22.722708151" Sep 10 22:05:21.167338 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:21.167288 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-96t46" podStartSLOduration=1.272566588 podStartE2EDuration="21.167270048s" totalImagesPullingTime="19.89470346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-10 22:05:21.166729918 +0000 UTC m=+22.736219882" watchObservedRunningTime="2026-09-10 22:05:21.167270048 +0000 UTC m=+22.736760012" Sep 10 22:05:21.931352 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:21.931323 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:21.931552 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:21.931430 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6nqn" podUID="673499d4-211d-40e2-8a3f-95c74bc1d0f3" Sep 10 22:05:22.931521 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:22.931486 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:05:22.931961 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:22.931605 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wfb5h" podUID="493e7732-8863-4f0f-b93e-bce47d8cf3de" Sep 10 22:05:22.931961 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:22.931663 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:05:22.931961 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:22.931786 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp72v" podUID="d748c035-3205-4b2f-8b7e-a229ff50e466" Sep 10 22:05:23.135817 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:23.135780 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" event={"ID":"6d6ce980-2a0b-474d-afe7-a72bd2e04f57","Type":"ContainerStarted","Data":"e817f30b5ef9318027ae2e37df612cdc149730e83d2ae6cd21f4cb9f5b2b8304"} Sep 10 22:05:23.139597 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:23.139569 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5j226" event={"ID":"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121","Type":"ContainerStarted","Data":"f580f0564e10b9472be84361605f910b70e47c379c7d9dc6ae1d38dd239c9a1c"} Sep 10 22:05:23.160559 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:23.160508 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7lbpg" podStartSLOduration=2.176957463 podStartE2EDuration="23.16049136s" totalImagesPullingTime="20.983533897s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:05:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:05:23.159934032 +0000 UTC m=+24.729423997" watchObservedRunningTime="2026-09-10 22:05:23.16049136 +0000 UTC m=+24.729981323" Sep 10 22:05:23.547968 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:23.547929 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-96t46" Sep 10 22:05:23.548804 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:23.548778 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-96t46" Sep 10 22:05:23.931035 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:23.931000 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:23.931259 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:23.931131 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6nqn" podUID="673499d4-211d-40e2-8a3f-95c74bc1d0f3" Sep 10 22:05:24.142454 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:24.142425 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-96t46" Sep 10 22:05:24.142948 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:24.142819 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-96t46" Sep 10 22:05:24.931879 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:24.931621 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:05:24.932051 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:24.931648 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:05:24.932051 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:24.932021 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp72v" podUID="d748c035-3205-4b2f-8b7e-a229ff50e466" Sep 10 22:05:24.932193 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:24.932103 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wfb5h" podUID="493e7732-8863-4f0f-b93e-bce47d8cf3de" Sep 10 22:05:25.145856 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:25.145820 2564 generic.go:356] "Generic (PLEG): container finished" podID="02564ad8-b933-4bc4-9e94-8995c093b6b0" containerID="758d269ee5714d6efb39b854e91691df75da77deea850948bbbc205546350a23" exitCode=0 Sep 10 22:05:25.146275 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:25.145916 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzxzf" event={"ID":"02564ad8-b933-4bc4-9e94-8995c093b6b0","Type":"ContainerDied","Data":"758d269ee5714d6efb39b854e91691df75da77deea850948bbbc205546350a23"} Sep 10 22:05:25.149953 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:25.149925 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5j226" event={"ID":"a7cb2d62-f2e4-4b3e-857e-908d0e1fd121","Type":"ContainerStarted","Data":"77f620b66fbde3f5e7307316679b8f3726aae31991ccc0a2257fc69f2aadee23"} Sep 10 22:05:25.150216 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:25.150199 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:05:25.150275 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:25.150259 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:05:25.164680 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:25.164654 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:05:25.164822 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:25.164757 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:05:25.203735 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:25.203686 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5j226" podStartSLOduration=7.200385627 podStartE2EDuration="26.203668221s" totalImagesPullingTime="19.003282594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:04:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:05:25.202411446 +0000 UTC m=+26.771901409" watchObservedRunningTime="2026-09-10 22:05:25.203668221 +0000 UTC m=+26.773158185" Sep 10 22:05:25.931139 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:25.931105 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:25.931285 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:25.931262 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6nqn" podUID="673499d4-211d-40e2-8a3f-95c74bc1d0f3" Sep 10 22:05:26.154388 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:26.154354 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzxzf" event={"ID":"02564ad8-b933-4bc4-9e94-8995c093b6b0","Type":"ContainerStarted","Data":"6cf1df038c79033b7d61aa82464cee4f1f48e426f77d6666e32c5320e1cddaf8"} Sep 10 22:05:26.155082 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:26.154398 2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 10 22:05:26.771708 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:26.771663 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m6nqn"] Sep 10 22:05:26.771892 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:26.771859 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:26.772030 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:26.772004 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6nqn" podUID="673499d4-211d-40e2-8a3f-95c74bc1d0f3" Sep 10 22:05:26.772425 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:26.772402 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wfb5h"] Sep 10 22:05:26.772514 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:26.772489 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:05:26.772592 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:26.772573 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wfb5h" podUID="493e7732-8863-4f0f-b93e-bce47d8cf3de" Sep 10 22:05:26.772980 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:26.772960 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fp72v"] Sep 10 22:05:26.773057 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:26.773046 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:05:26.773163 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:26.773142 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp72v" podUID="d748c035-3205-4b2f-8b7e-a229ff50e466" Sep 10 22:05:27.159076 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:27.159036 2564 generic.go:356] "Generic (PLEG): container finished" podID="02564ad8-b933-4bc4-9e94-8995c093b6b0" containerID="6cf1df038c79033b7d61aa82464cee4f1f48e426f77d6666e32c5320e1cddaf8" exitCode=0 Sep 10 22:05:27.159533 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:27.159117 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzxzf" event={"ID":"02564ad8-b933-4bc4-9e94-8995c093b6b0","Type":"ContainerDied","Data":"6cf1df038c79033b7d61aa82464cee4f1f48e426f77d6666e32c5320e1cddaf8"} Sep 10 22:05:27.159533 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:27.159201 2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 10 22:05:27.931648 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:27.931607 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:27.931829 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:27.931717 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6nqn" podUID="673499d4-211d-40e2-8a3f-95c74bc1d0f3" Sep 10 22:05:28.163851 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:28.163819 2564 generic.go:356] "Generic (PLEG): container finished" podID="02564ad8-b933-4bc4-9e94-8995c093b6b0" containerID="a816073f56a5d0057592a8b3a59be2a163f29592803d52597b88a55f65c36266" exitCode=0 Sep 10 22:05:28.164468 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:28.163918 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzxzf" event={"ID":"02564ad8-b933-4bc4-9e94-8995c093b6b0","Type":"ContainerDied","Data":"a816073f56a5d0057592a8b3a59be2a163f29592803d52597b88a55f65c36266"} Sep 10 22:05:28.931608 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:28.931575 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:05:28.931787 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:28.931704 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp72v" podUID="d748c035-3205-4b2f-8b7e-a229ff50e466" Sep 10 22:05:28.931787 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:28.931749 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:05:28.931933 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:28.931847 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wfb5h" podUID="493e7732-8863-4f0f-b93e-bce47d8cf3de" Sep 10 22:05:29.931041 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:29.930834 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:29.931407 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:29.931112 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6nqn" podUID="673499d4-211d-40e2-8a3f-95c74bc1d0f3" Sep 10 22:05:30.113982 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:30.113945 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:05:30.114224 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:30.114202 2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 10 22:05:30.126555 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:30.126502 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-5j226" podUID="a7cb2d62-f2e4-4b3e-857e-908d0e1fd121" containerName="ovnkube-controller" probeResult="failure" output="" Sep 10 22:05:30.135994 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:30.135960 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-5j226" podUID="a7cb2d62-f2e4-4b3e-857e-908d0e1fd121" containerName="ovnkube-controller" probeResult="failure" output="" Sep 10 22:05:30.931152 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:30.930886 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:05:30.931152 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:30.931031 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wfb5h" podUID="493e7732-8863-4f0f-b93e-bce47d8cf3de" Sep 10 22:05:30.931152 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:30.931047 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:05:30.931721 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:30.931206 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp72v" podUID="d748c035-3205-4b2f-8b7e-a229ff50e466" Sep 10 22:05:31.586631 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:31.586598 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs\") pod \"network-metrics-daemon-fp72v\" (UID: \"d748c035-3205-4b2f-8b7e-a229ff50e466\") " pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:05:31.586789 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:31.586707 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 22:05:31.586789 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:31.586759 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs podName:d748c035-3205-4b2f-8b7e-a229ff50e466 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:03.586746059 +0000 UTC m=+65.156236014 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs") pod "network-metrics-daemon-fp72v" (UID: "d748c035-3205-4b2f-8b7e-a229ff50e466") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 22:05:31.687456 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:31.687426 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/673499d4-211d-40e2-8a3f-95c74bc1d0f3-original-pull-secret\") pod \"global-pull-secret-syncer-m6nqn\" (UID: \"673499d4-211d-40e2-8a3f-95c74bc1d0f3\") " pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:31.687617 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:31.687532 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 22:05:31.687617 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:31.687583 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/673499d4-211d-40e2-8a3f-95c74bc1d0f3-original-pull-secret podName:673499d4-211d-40e2-8a3f-95c74bc1d0f3 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:03.68756993 +0000 UTC m=+65.257059873 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/673499d4-211d-40e2-8a3f-95c74bc1d0f3-original-pull-secret") pod "global-pull-secret-syncer-m6nqn" (UID: "673499d4-211d-40e2-8a3f-95c74bc1d0f3") : object "kube-system"/"original-pull-secret" not registered Sep 10 22:05:31.788228 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:31.788196 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-765sx\" (UniqueName: \"kubernetes.io/projected/493e7732-8863-4f0f-b93e-bce47d8cf3de-kube-api-access-765sx\") pod \"network-check-target-wfb5h\" (UID: \"493e7732-8863-4f0f-b93e-bce47d8cf3de\") " pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:05:31.788385 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:31.788306 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 22:05:31.788385 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:31.788318 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 22:05:31.788385 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:31.788327 2564 projected.go:196] Error preparing data for projected volume kube-api-access-765sx for pod openshift-network-diagnostics/network-check-target-wfb5h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 22:05:31.788385 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:31.788369 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/493e7732-8863-4f0f-b93e-bce47d8cf3de-kube-api-access-765sx podName:493e7732-8863-4f0f-b93e-bce47d8cf3de nodeName:}" failed. No retries permitted until 2026-09-10 22:06:03.788356588 +0000 UTC m=+65.357846543 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-765sx" (UniqueName: "kubernetes.io/projected/493e7732-8863-4f0f-b93e-bce47d8cf3de-kube-api-access-765sx") pod "network-check-target-wfb5h" (UID: "493e7732-8863-4f0f-b93e-bce47d8cf3de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 22:05:31.930695 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:31.930662 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:31.930858 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:31.930769 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m6nqn" podUID="673499d4-211d-40e2-8a3f-95c74bc1d0f3" Sep 10 22:05:32.935440 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:32.935409 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:05:32.935440 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:32.935443 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:05:32.935963 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:32.935541 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp72v" podUID="d748c035-3205-4b2f-8b7e-a229ff50e466" Sep 10 22:05:32.935963 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:32.935665 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wfb5h" podUID="493e7732-8863-4f0f-b93e-bce47d8cf3de" Sep 10 22:05:33.201974 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.201939 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-148.ec2.internal" event="NodeReady" Sep 10 22:05:33.202175 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.202103 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 22:05:33.254527 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.254492 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7bb7f54487-lqtc8"] Sep 10 22:05:33.275212 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.275178 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lqltl"] Sep 10 22:05:33.275381 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.275276 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:33.278157 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.278132 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 10 22:05:33.278275 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.278197 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7vdxb\"" Sep 10 22:05:33.278455 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.278437 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 10 22:05:33.278689 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.278614 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 10 22:05:33.285793 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.285769 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 10 22:05:33.291122 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.291101 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mp766"] Sep 10 22:05:33.291248 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.291231 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lqltl" Sep 10 22:05:33.294686 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.294647 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 22:05:33.294959 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.294814 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mb5j2\"" Sep 10 22:05:33.294959 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.294825 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 22:05:33.313227 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.313201 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lqltl"] Sep 10 22:05:33.313384 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.313248 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mp766"] Sep 10 22:05:33.313384 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.313269 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bb7f54487-lqtc8"] Sep 10 22:05:33.313384 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.313329 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mp766" Sep 10 22:05:33.316349 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.316327 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 22:05:33.316604 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.316583 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wxdc9\"" Sep 10 22:05:33.316850 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.316827 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 22:05:33.319311 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.319288 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 22:05:33.322930 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.322889 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96t46_4f2170c9-cacb-47f3-8a42-b80039deefef/konnectivity-agent/0.log" limit=1024 Sep 10 22:05:33.400827 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.400797 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:33.400827 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.400831 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0676c0bc-49e3-416a-97eb-70199abcf210-metrics-tls\") pod \"dns-default-lqltl\" (UID: \"0676c0bc-49e3-416a-97eb-70199abcf210\") " pod="openshift-dns/dns-default-lqltl" Sep 10 22:05:33.401086 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.400849 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb86s\" (UniqueName: \"kubernetes.io/projected/0676c0bc-49e3-416a-97eb-70199abcf210-kube-api-access-mb86s\") pod \"dns-default-lqltl\" (UID: \"0676c0bc-49e3-416a-97eb-70199abcf210\") " pod="openshift-dns/dns-default-lqltl" Sep 10 22:05:33.401086 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.400871 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m68r\" (UniqueName: \"kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-kube-api-access-4m68r\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:33.401086 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.400928 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-ca-trust-extracted\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:33.401086 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.400946 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-trusted-ca\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:33.401086 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.401014 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-installation-pull-secrets\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:33.401272 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.401120 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/643143f3-0560-433b-9fca-05ef72574ee2-cert\") pod \"ingress-canary-mp766\" (UID: \"643143f3-0560-433b-9fca-05ef72574ee2\") " pod="openshift-ingress-canary/ingress-canary-mp766" Sep 10 22:05:33.401272 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.401156 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-certificates\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:33.401272 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.401171 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0676c0bc-49e3-416a-97eb-70199abcf210-config-volume\") pod \"dns-default-lqltl\" (UID: \"0676c0bc-49e3-416a-97eb-70199abcf210\") " pod="openshift-dns/dns-default-lqltl" Sep 10 22:05:33.401272 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.401206 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-image-registry-private-configuration\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:33.401272 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.401228 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-bound-sa-token\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:33.401272 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.401249 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0676c0bc-49e3-416a-97eb-70199abcf210-tmp-dir\") pod \"dns-default-lqltl\" (UID: \"0676c0bc-49e3-416a-97eb-70199abcf210\") " pod="openshift-dns/dns-default-lqltl" Sep 10 22:05:33.401272 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.401269 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69m9h\" (UniqueName: \"kubernetes.io/projected/643143f3-0560-433b-9fca-05ef72574ee2-kube-api-access-69m9h\") pod \"ingress-canary-mp766\" (UID: \"643143f3-0560-433b-9fca-05ef72574ee2\") " pod="openshift-ingress-canary/ingress-canary-mp766" Sep 10 22:05:33.463783 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.463700 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96t46_4f2170c9-cacb-47f3-8a42-b80039deefef/konnectivity-agent/0.log" limit=1024 Sep 10 22:05:33.501686 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.501638 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-installation-pull-secrets\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:33.501686 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.501692 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/643143f3-0560-433b-9fca-05ef72574ee2-cert\") pod \"ingress-canary-mp766\" (UID: \"643143f3-0560-433b-9fca-05ef72574ee2\") " pod="openshift-ingress-canary/ingress-canary-mp766" Sep 10 22:05:33.501947 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.501724 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-certificates\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:33.501947 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.501749 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0676c0bc-49e3-416a-97eb-70199abcf210-config-volume\") pod \"dns-default-lqltl\" (UID: \"0676c0bc-49e3-416a-97eb-70199abcf210\") " pod="openshift-dns/dns-default-lqltl" Sep 10 22:05:33.501947 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.501802 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-image-registry-private-configuration\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:33.501947 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.501846 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-bound-sa-token\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:33.501947 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.501866 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0676c0bc-49e3-416a-97eb-70199abcf210-tmp-dir\") pod \"dns-default-lqltl\" (UID: \"0676c0bc-49e3-416a-97eb-70199abcf210\") " pod="openshift-dns/dns-default-lqltl" Sep 10 22:05:33.501947 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.501888 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69m9h\" (UniqueName: \"kubernetes.io/projected/643143f3-0560-433b-9fca-05ef72574ee2-kube-api-access-69m9h\") pod \"ingress-canary-mp766\" (UID: \"643143f3-0560-433b-9fca-05ef72574ee2\") " pod="openshift-ingress-canary/ingress-canary-mp766" Sep 10 22:05:33.501947 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.501933 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:33.502181 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.501958 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0676c0bc-49e3-416a-97eb-70199abcf210-metrics-tls\") pod \"dns-default-lqltl\" (UID: \"0676c0bc-49e3-416a-97eb-70199abcf210\") " pod="openshift-dns/dns-default-lqltl" Sep 10 22:05:33.502181 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.501983 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb86s\" (UniqueName: \"kubernetes.io/projected/0676c0bc-49e3-416a-97eb-70199abcf210-kube-api-access-mb86s\") pod \"dns-default-lqltl\" (UID: \"0676c0bc-49e3-416a-97eb-70199abcf210\") " pod="openshift-dns/dns-default-lqltl" Sep 10 22:05:33.502181 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.502013 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m68r\" (UniqueName: \"kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-kube-api-access-4m68r\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:33.502181 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.502056 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-ca-trust-extracted\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:33.502181 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.502078 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-trusted-ca\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:33.503137 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.503110 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-certificates\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:33.503280 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.503148 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-trusted-ca\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:33.503280 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:33.503247 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 22:05:33.503280 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:33.503262 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bb7f54487-lqtc8: secret "image-registry-tls" not found Sep 10 22:05:33.503406 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.503294 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0676c0bc-49e3-416a-97eb-70199abcf210-tmp-dir\") pod \"dns-default-lqltl\" (UID: \"0676c0bc-49e3-416a-97eb-70199abcf210\") " pod="openshift-dns/dns-default-lqltl" Sep 10 22:05:33.503449 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:33.503422 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls podName:bd8bd0ce-a4ba-4134-89aa-224eace7e64d nodeName:}" failed. No retries permitted until 2026-09-10 22:05:34.003394711 +0000 UTC m=+35.572884655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls") pod "image-registry-7bb7f54487-lqtc8" (UID: "bd8bd0ce-a4ba-4134-89aa-224eace7e64d") : secret "image-registry-tls" not found Sep 10 22:05:33.503449 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:33.503442 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 22:05:33.503546 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:33.503505 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0676c0bc-49e3-416a-97eb-70199abcf210-metrics-tls podName:0676c0bc-49e3-416a-97eb-70199abcf210 nodeName:}" failed. No retries permitted until 2026-09-10 22:05:34.003486367 +0000 UTC m=+35.572976316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0676c0bc-49e3-416a-97eb-70199abcf210-metrics-tls") pod "dns-default-lqltl" (UID: "0676c0bc-49e3-416a-97eb-70199abcf210") : secret "dns-default-metrics-tls" not found Sep 10 22:05:33.503597 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.503554 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0676c0bc-49e3-416a-97eb-70199abcf210-config-volume\") pod \"dns-default-lqltl\" (UID: \"0676c0bc-49e3-416a-97eb-70199abcf210\") " pod="openshift-dns/dns-default-lqltl" Sep 10 22:05:33.503799 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.503777 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-ca-trust-extracted\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:33.506749 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.506725 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-installation-pull-secrets\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:33.506881 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.506726 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-image-registry-private-configuration\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:33.507179 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.507160 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/643143f3-0560-433b-9fca-05ef72574ee2-cert\") pod \"ingress-canary-mp766\" (UID: \"643143f3-0560-433b-9fca-05ef72574ee2\") " pod="openshift-ingress-canary/ingress-canary-mp766" Sep 10 22:05:33.520991 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.520958 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb86s\" (UniqueName: \"kubernetes.io/projected/0676c0bc-49e3-416a-97eb-70199abcf210-kube-api-access-mb86s\") pod \"dns-default-lqltl\" (UID: \"0676c0bc-49e3-416a-97eb-70199abcf210\") " pod="openshift-dns/dns-default-lqltl" Sep 10 22:05:33.521138 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.521081 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-bound-sa-token\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:33.521138 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.521104 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m68r\" (UniqueName: \"kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-kube-api-access-4m68r\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:33.533947 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.533917 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69m9h\" (UniqueName: \"kubernetes.io/projected/643143f3-0560-433b-9fca-05ef72574ee2-kube-api-access-69m9h\") pod \"ingress-canary-mp766\" (UID: \"643143f3-0560-433b-9fca-05ef72574ee2\") " pod="openshift-ingress-canary/ingress-canary-mp766" Sep 10 22:05:33.623212 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.623173 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mp766" Sep 10 22:05:33.689289 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.689249 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96t46_4f2170c9-cacb-47f3-8a42-b80039deefef/konnectivity-agent/0.log" limit=1024 Sep 10 22:05:33.831111 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.831021 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96t46_4f2170c9-cacb-47f3-8a42-b80039deefef/konnectivity-agent/0.log" limit=1024 Sep 10 22:05:33.931636 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.931600 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:05:33.934646 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:33.934623 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 22:05:34.006871 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:34.006835 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:34.006871 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:34.006876 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0676c0bc-49e3-416a-97eb-70199abcf210-metrics-tls\") pod \"dns-default-lqltl\" (UID: \"0676c0bc-49e3-416a-97eb-70199abcf210\") " pod="openshift-dns/dns-default-lqltl" Sep 10 22:05:34.007362 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:34.007009 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 22:05:34.007362 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:34.007034 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bb7f54487-lqtc8: secret "image-registry-tls" not found Sep 10 22:05:34.007362 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:34.007038 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 22:05:34.007362 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:34.007099 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls podName:bd8bd0ce-a4ba-4134-89aa-224eace7e64d nodeName:}" failed. No retries permitted until 2026-09-10 22:05:35.007079626 +0000 UTC m=+36.576569591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls") pod "image-registry-7bb7f54487-lqtc8" (UID: "bd8bd0ce-a4ba-4134-89aa-224eace7e64d") : secret "image-registry-tls" not found Sep 10 22:05:34.007362 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:34.007114 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0676c0bc-49e3-416a-97eb-70199abcf210-metrics-tls podName:0676c0bc-49e3-416a-97eb-70199abcf210 nodeName:}" failed. No retries permitted until 2026-09-10 22:05:35.007107513 +0000 UTC m=+36.576597455 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0676c0bc-49e3-416a-97eb-70199abcf210-metrics-tls") pod "dns-default-lqltl" (UID: "0676c0bc-49e3-416a-97eb-70199abcf210") : secret "dns-default-metrics-tls" not found Sep 10 22:05:34.580809 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:34.578471 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mp766"] Sep 10 22:05:34.584087 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:05:34.584053 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643143f3_0560_433b_9fca_05ef72574ee2.slice/crio-d1834870652e5967ccd84b471b23296d60b17c18c55b5a426fd666aa94023308 WatchSource:0}: Error finding container d1834870652e5967ccd84b471b23296d60b17c18c55b5a426fd666aa94023308: Status 404 returned error can't find the container with id d1834870652e5967ccd84b471b23296d60b17c18c55b5a426fd666aa94023308 Sep 10 22:05:34.934423 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:34.934393 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:05:34.934586 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:34.934550 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:05:34.937638 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:34.937619 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 22:05:34.937638 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:34.937626 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dmqhm\"" Sep 10 22:05:34.937818 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:34.937620 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 22:05:34.937818 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:34.937690 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 22:05:34.938774 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:34.938756 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8m774\"" Sep 10 22:05:35.015238 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:35.015200 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:35.015760 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:35.015311 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 22:05:35.015760 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:35.015324 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bb7f54487-lqtc8: secret "image-registry-tls" not found Sep 10 22:05:35.015760 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:35.015343 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0676c0bc-49e3-416a-97eb-70199abcf210-metrics-tls\") pod \"dns-default-lqltl\" (UID: \"0676c0bc-49e3-416a-97eb-70199abcf210\") " pod="openshift-dns/dns-default-lqltl" Sep 10 22:05:35.015760 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:35.015381 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls podName:bd8bd0ce-a4ba-4134-89aa-224eace7e64d nodeName:}" failed. No retries permitted until 2026-09-10 22:05:37.015365321 +0000 UTC m=+38.584855293 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls") pod "image-registry-7bb7f54487-lqtc8" (UID: "bd8bd0ce-a4ba-4134-89aa-224eace7e64d") : secret "image-registry-tls" not found Sep 10 22:05:35.015760 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:35.015440 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 22:05:35.015760 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:35.015491 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0676c0bc-49e3-416a-97eb-70199abcf210-metrics-tls podName:0676c0bc-49e3-416a-97eb-70199abcf210 nodeName:}" failed. No retries permitted until 2026-09-10 22:05:37.015477788 +0000 UTC m=+38.584967744 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0676c0bc-49e3-416a-97eb-70199abcf210-metrics-tls") pod "dns-default-lqltl" (UID: "0676c0bc-49e3-416a-97eb-70199abcf210") : secret "dns-default-metrics-tls" not found Sep 10 22:05:35.184587 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:35.184555 2564 generic.go:356] "Generic (PLEG): container finished" podID="02564ad8-b933-4bc4-9e94-8995c093b6b0" containerID="fc0375b56ad5f87e3995d3e160c36336fb4653a4fba693af079a88a6b7275b3a" exitCode=0 Sep 10 22:05:35.184768 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:35.184635 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzxzf" event={"ID":"02564ad8-b933-4bc4-9e94-8995c093b6b0","Type":"ContainerDied","Data":"fc0375b56ad5f87e3995d3e160c36336fb4653a4fba693af079a88a6b7275b3a"} Sep 10 22:05:35.185960 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:35.185937 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mp766" event={"ID":"643143f3-0560-433b-9fca-05ef72574ee2","Type":"ContainerStarted","Data":"d1834870652e5967ccd84b471b23296d60b17c18c55b5a426fd666aa94023308"} Sep 10 22:05:36.191766 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:36.191739 2564 generic.go:356] "Generic (PLEG): container finished" podID="02564ad8-b933-4bc4-9e94-8995c093b6b0" containerID="39782d3883830225edc95f04f6680fda9ec386f847bc57e7a25585eba1a622c9" exitCode=0 Sep 10 22:05:36.192148 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:36.191795 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzxzf" event={"ID":"02564ad8-b933-4bc4-9e94-8995c093b6b0","Type":"ContainerDied","Data":"39782d3883830225edc95f04f6680fda9ec386f847bc57e7a25585eba1a622c9"} Sep 10 22:05:37.028192 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:37.028159 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:37.028367 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:37.028207 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0676c0bc-49e3-416a-97eb-70199abcf210-metrics-tls\") pod \"dns-default-lqltl\" (UID: \"0676c0bc-49e3-416a-97eb-70199abcf210\") " pod="openshift-dns/dns-default-lqltl" Sep 10 22:05:37.028367 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:37.028302 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 22:05:37.028367 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:37.028316 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 22:05:37.028367 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:37.028338 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bb7f54487-lqtc8: secret "image-registry-tls" not found Sep 10 22:05:37.028559 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:37.028382 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0676c0bc-49e3-416a-97eb-70199abcf210-metrics-tls podName:0676c0bc-49e3-416a-97eb-70199abcf210 nodeName:}" failed. No retries permitted until 2026-09-10 22:05:41.028361685 +0000 UTC m=+42.597851648 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0676c0bc-49e3-416a-97eb-70199abcf210-metrics-tls") pod "dns-default-lqltl" (UID: "0676c0bc-49e3-416a-97eb-70199abcf210") : secret "dns-default-metrics-tls" not found Sep 10 22:05:37.028559 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:37.028441 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls podName:bd8bd0ce-a4ba-4134-89aa-224eace7e64d nodeName:}" failed. No retries permitted until 2026-09-10 22:05:41.028422379 +0000 UTC m=+42.597912324 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls") pod "image-registry-7bb7f54487-lqtc8" (UID: "bd8bd0ce-a4ba-4134-89aa-224eace7e64d") : secret "image-registry-tls" not found Sep 10 22:05:37.196964 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:37.196930 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzxzf" event={"ID":"02564ad8-b933-4bc4-9e94-8995c093b6b0","Type":"ContainerStarted","Data":"36c044ddad113f5e042da09d6de34bb5913755b0ee7a316133f4bc6b60d91354"} Sep 10 22:05:37.228577 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:37.226079 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lzxzf" podStartSLOduration=6.6886881670000005 podStartE2EDuration="38.226058591s" totalImagesPullingTime="31.537370424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:04:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:05:37.225762214 +0000 UTC m=+38.795252160" watchObservedRunningTime="2026-09-10 22:05:37.226058591 +0000 UTC m=+38.795548555" Sep 10 22:05:37.347115 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:37.347024 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96t46_4f2170c9-cacb-47f3-8a42-b80039deefef/konnectivity-agent/0.log" limit=1024 Sep 10 22:05:37.446160 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:37.446111 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96t46_4f2170c9-cacb-47f3-8a42-b80039deefef/konnectivity-agent/0.log" limit=1024 Sep 10 22:05:38.200543 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:38.200499 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mp766" event={"ID":"643143f3-0560-433b-9fca-05ef72574ee2","Type":"ContainerStarted","Data":"8f048faa50c7ab48f5e4abd8e08fb78c97c3ca053c6588b5abeff6db1431c076"} Sep 10 22:05:38.217685 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:38.217631 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mp766" podStartSLOduration=2.554583497 podStartE2EDuration="5.217614481s" totalImagesPullingTime="2.663030984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:05:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:05:38.217031584 +0000 UTC m=+39.786521547" watchObservedRunningTime="2026-09-10 22:05:38.217614481 +0000 UTC m=+39.787104483" Sep 10 22:05:40.463960 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:40.463918 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-96t46_4f2170c9-cacb-47f3-8a42-b80039deefef/konnectivity-agent/0.log" limit=1024 Sep 10 22:05:41.055866 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:41.055834 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:41.056046 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:41.055876 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0676c0bc-49e3-416a-97eb-70199abcf210-metrics-tls\") pod \"dns-default-lqltl\" (UID: \"0676c0bc-49e3-416a-97eb-70199abcf210\") " pod="openshift-dns/dns-default-lqltl" Sep 10 22:05:41.056046 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:41.055984 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 22:05:41.056046 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:41.056003 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bb7f54487-lqtc8: secret "image-registry-tls" not found Sep 10 22:05:41.056046 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:41.056027 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 22:05:41.056177 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:41.056056 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls podName:bd8bd0ce-a4ba-4134-89aa-224eace7e64d nodeName:}" failed. No retries permitted until 2026-09-10 22:05:49.056042096 +0000 UTC m=+50.625532038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls") pod "image-registry-7bb7f54487-lqtc8" (UID: "bd8bd0ce-a4ba-4134-89aa-224eace7e64d") : secret "image-registry-tls" not found Sep 10 22:05:41.056177 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:41.056084 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0676c0bc-49e3-416a-97eb-70199abcf210-metrics-tls podName:0676c0bc-49e3-416a-97eb-70199abcf210 nodeName:}" failed. No retries permitted until 2026-09-10 22:05:49.05606419 +0000 UTC m=+50.625554153 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0676c0bc-49e3-416a-97eb-70199abcf210-metrics-tls") pod "dns-default-lqltl" (UID: "0676c0bc-49e3-416a-97eb-70199abcf210") : secret "dns-default-metrics-tls" not found Sep 10 22:05:49.111861 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:49.111833 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:05:49.111861 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:05:49.111863 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0676c0bc-49e3-416a-97eb-70199abcf210-metrics-tls\") pod \"dns-default-lqltl\" (UID: \"0676c0bc-49e3-416a-97eb-70199abcf210\") " pod="openshift-dns/dns-default-lqltl" Sep 10 22:05:49.112390 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:49.111974 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 22:05:49.112390 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:49.111996 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 22:05:49.112390 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:49.112012 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bb7f54487-lqtc8: secret "image-registry-tls" not found Sep 10 22:05:49.112390 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:49.112015 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0676c0bc-49e3-416a-97eb-70199abcf210-metrics-tls podName:0676c0bc-49e3-416a-97eb-70199abcf210 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:05.112003033 +0000 UTC m=+66.681492975 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0676c0bc-49e3-416a-97eb-70199abcf210-metrics-tls") pod "dns-default-lqltl" (UID: "0676c0bc-49e3-416a-97eb-70199abcf210") : secret "dns-default-metrics-tls" not found Sep 10 22:05:49.112390 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:05:49.112096 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls podName:bd8bd0ce-a4ba-4134-89aa-224eace7e64d nodeName:}" failed. No retries permitted until 2026-09-10 22:06:05.112064966 +0000 UTC m=+66.681554923 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls") pod "image-registry-7bb7f54487-lqtc8" (UID: "bd8bd0ce-a4ba-4134-89aa-224eace7e64d") : secret "image-registry-tls" not found Sep 10 22:06:00.135253 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:00.135223 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5j226" Sep 10 22:06:03.453700 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.453667 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-mcmvz"] Sep 10 22:06:03.459361 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.459339 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-mcmvz" Sep 10 22:06:03.461937 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.461918 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vg7rm\"" Sep 10 22:06:03.473289 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.473269 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-mcmvz"] Sep 10 22:06:03.531963 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.531942 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-fpgjg"] Sep 10 22:06:03.535149 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.535135 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fpgjg" Sep 10 22:06:03.542116 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.542098 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 22:06:03.542254 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.542232 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 10 22:06:03.542360 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.542314 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hrtfz\"" Sep 10 22:06:03.542360 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.542342 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 10 22:06:03.542472 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.542107 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 22:06:03.542952 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.542934 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-8ssql"] Sep 10 22:06:03.546253 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.546238 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-hq6mn"] Sep 10 22:06:03.546342 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.546330 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8ssql" Sep 10 22:06:03.549250 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.549235 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-hq6mn" Sep 10 22:06:03.552565 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.552548 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 10 22:06:03.552676 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.552644 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-zm2d7\"" Sep 10 22:06:03.552730 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.552683 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 10 22:06:03.552730 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.552716 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 10 22:06:03.552853 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.552753 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 10 22:06:03.552853 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.552772 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 22:06:03.552853 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.552829 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 10 22:06:03.553003 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.552919 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-z6bpw\"" Sep 10 22:06:03.553003 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.552992 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 22:06:03.559241 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.559210 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 10 22:06:03.565615 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.565596 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-fpgjg"] Sep 10 22:06:03.567553 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.567535 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 10 22:06:03.576273 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.576252 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-hq6mn"] Sep 10 22:06:03.576888 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.576869 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-8ssql"] Sep 10 22:06:03.620788 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.620768 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs\") pod \"network-metrics-daemon-fp72v\" (UID: \"d748c035-3205-4b2f-8b7e-a229ff50e466\") " pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:06:03.620893 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.620816 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q46xk\" (UniqueName: \"kubernetes.io/projected/e9451128-97b3-4feb-b391-c2817898aa2c-kube-api-access-q46xk\") pod \"network-check-source-6f8b768455-mcmvz\" (UID: \"e9451128-97b3-4feb-b391-c2817898aa2c\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-mcmvz" Sep 10 22:06:03.620893 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.620866 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1086cb8d-b834-4e42-87df-6db5b0bd9037-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fpgjg\" (UID: \"1086cb8d-b834-4e42-87df-6db5b0bd9037\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fpgjg" Sep 10 22:06:03.621011 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.620914 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1086cb8d-b834-4e42-87df-6db5b0bd9037-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-fpgjg\" (UID: \"1086cb8d-b834-4e42-87df-6db5b0bd9037\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fpgjg" Sep 10 22:06:03.621061 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.621044 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7jzz\" (UniqueName: \"kubernetes.io/projected/1086cb8d-b834-4e42-87df-6db5b0bd9037-kube-api-access-s7jzz\") pod \"cluster-monitoring-operator-6cd954b897-fpgjg\" (UID: \"1086cb8d-b834-4e42-87df-6db5b0bd9037\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fpgjg" Sep 10 22:06:03.623067 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.623054 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 22:06:03.631827 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:03.631809 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 10 22:06:03.631921 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:03.631896 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs podName:d748c035-3205-4b2f-8b7e-a229ff50e466 nodeName:}" failed. No retries permitted until 2026-09-10 22:07:07.63188218 +0000 UTC m=+129.201372142 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs") pod "network-metrics-daemon-fp72v" (UID: "d748c035-3205-4b2f-8b7e-a229ff50e466") : secret "metrics-daemon-secret" not found Sep 10 22:06:03.636176 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.636158 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-5fl5b"] Sep 10 22:06:03.639218 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.639204 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-5fl5b" Sep 10 22:06:03.642508 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.642490 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-x8cdz\"" Sep 10 22:06:03.642624 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.642496 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 10 22:06:03.642702 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.642688 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 10 22:06:03.654518 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.654499 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-5fl5b"] Sep 10 22:06:03.721394 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.721339 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0bc3a376-76d5-44b1-b9eb-0b988cd4b1dd-serving-cert\") pod \"service-ca-operator-cf7c776b7-8ssql\" (UID: \"0bc3a376-76d5-44b1-b9eb-0b988cd4b1dd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8ssql" Sep 10 22:06:03.721394 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.721384 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2b50e35-7ca5-420b-a788-73f839011390-trusted-ca-bundle\") pod \"insights-operator-5844b856db-hq6mn\" (UID: \"c2b50e35-7ca5-420b-a788-73f839011390\") " pod="openshift-insights/insights-operator-5844b856db-hq6mn" Sep 10 22:06:03.721511 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.721409 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q46xk\" (UniqueName: \"kubernetes.io/projected/e9451128-97b3-4feb-b391-c2817898aa2c-kube-api-access-q46xk\") pod \"network-check-source-6f8b768455-mcmvz\" (UID: \"e9451128-97b3-4feb-b391-c2817898aa2c\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-mcmvz" Sep 10 22:06:03.721511 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.721470 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqms6\" (UniqueName: \"kubernetes.io/projected/c2b50e35-7ca5-420b-a788-73f839011390-kube-api-access-pqms6\") pod \"insights-operator-5844b856db-hq6mn\" (UID: \"c2b50e35-7ca5-420b-a788-73f839011390\") " pod="openshift-insights/insights-operator-5844b856db-hq6mn" Sep 10 22:06:03.721511 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.721504 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1086cb8d-b834-4e42-87df-6db5b0bd9037-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fpgjg\" (UID: \"1086cb8d-b834-4e42-87df-6db5b0bd9037\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fpgjg" Sep 10 22:06:03.721603 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.721525 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2b50e35-7ca5-420b-a788-73f839011390-tmp\") pod \"insights-operator-5844b856db-hq6mn\" (UID: \"c2b50e35-7ca5-420b-a788-73f839011390\") " pod="openshift-insights/insights-operator-5844b856db-hq6mn" Sep 10 22:06:03.721603 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.721566 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6wv4\" (UniqueName: \"kubernetes.io/projected/0bc3a376-76d5-44b1-b9eb-0b988cd4b1dd-kube-api-access-g6wv4\") pod \"service-ca-operator-cf7c776b7-8ssql\" (UID: \"0bc3a376-76d5-44b1-b9eb-0b988cd4b1dd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8ssql" Sep 10 22:06:03.721603 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.721590 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1086cb8d-b834-4e42-87df-6db5b0bd9037-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-fpgjg\" (UID: \"1086cb8d-b834-4e42-87df-6db5b0bd9037\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fpgjg" Sep 10 22:06:03.721691 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:03.721606 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 22:06:03.721691 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.721615 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0bc3a376-76d5-44b1-b9eb-0b988cd4b1dd-config\") pod \"service-ca-operator-cf7c776b7-8ssql\" (UID: \"0bc3a376-76d5-44b1-b9eb-0b988cd4b1dd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8ssql" Sep 10 22:06:03.721691 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:03.721659 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1086cb8d-b834-4e42-87df-6db5b0bd9037-cluster-monitoring-operator-tls podName:1086cb8d-b834-4e42-87df-6db5b0bd9037 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:04.221645667 +0000 UTC m=+65.791135638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1086cb8d-b834-4e42-87df-6db5b0bd9037-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fpgjg" (UID: "1086cb8d-b834-4e42-87df-6db5b0bd9037") : secret "cluster-monitoring-operator-tls" not found Sep 10 22:06:03.721786 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.721703 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/673499d4-211d-40e2-8a3f-95c74bc1d0f3-original-pull-secret\") pod \"global-pull-secret-syncer-m6nqn\" (UID: \"673499d4-211d-40e2-8a3f-95c74bc1d0f3\") " pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:06:03.721786 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.721744 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7jzz\" (UniqueName: \"kubernetes.io/projected/1086cb8d-b834-4e42-87df-6db5b0bd9037-kube-api-access-s7jzz\") pod \"cluster-monitoring-operator-6cd954b897-fpgjg\" (UID: \"1086cb8d-b834-4e42-87df-6db5b0bd9037\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fpgjg" Sep 10 22:06:03.721786 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.721766 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2b50e35-7ca5-420b-a788-73f839011390-service-ca-bundle\") pod \"insights-operator-5844b856db-hq6mn\" (UID: \"c2b50e35-7ca5-420b-a788-73f839011390\") " pod="openshift-insights/insights-operator-5844b856db-hq6mn" Sep 10 22:06:03.721935 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.721791 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c2b50e35-7ca5-420b-a788-73f839011390-snapshots\") pod \"insights-operator-5844b856db-hq6mn\" (UID: \"c2b50e35-7ca5-420b-a788-73f839011390\") " pod="openshift-insights/insights-operator-5844b856db-hq6mn" Sep 10 22:06:03.721935 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.721815 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2b50e35-7ca5-420b-a788-73f839011390-serving-cert\") pod \"insights-operator-5844b856db-hq6mn\" (UID: \"c2b50e35-7ca5-420b-a788-73f839011390\") " pod="openshift-insights/insights-operator-5844b856db-hq6mn" Sep 10 22:06:03.722248 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.722230 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1086cb8d-b834-4e42-87df-6db5b0bd9037-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-fpgjg\" (UID: \"1086cb8d-b834-4e42-87df-6db5b0bd9037\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fpgjg" Sep 10 22:06:03.726301 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.726285 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 22:06:03.734958 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.734934 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/673499d4-211d-40e2-8a3f-95c74bc1d0f3-original-pull-secret\") pod \"global-pull-secret-syncer-m6nqn\" (UID: \"673499d4-211d-40e2-8a3f-95c74bc1d0f3\") " pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:06:03.736124 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.736099 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 22:06:03.737564 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.737544 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7jzz\" (UniqueName: \"kubernetes.io/projected/1086cb8d-b834-4e42-87df-6db5b0bd9037-kube-api-access-s7jzz\") pod \"cluster-monitoring-operator-6cd954b897-fpgjg\" (UID: \"1086cb8d-b834-4e42-87df-6db5b0bd9037\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fpgjg" Sep 10 22:06:03.746868 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.746846 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 22:06:03.756490 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.756466 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q46xk\" (UniqueName: \"kubernetes.io/projected/e9451128-97b3-4feb-b391-c2817898aa2c-kube-api-access-q46xk\") pod \"network-check-source-6f8b768455-mcmvz\" (UID: \"e9451128-97b3-4feb-b391-c2817898aa2c\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-mcmvz" Sep 10 22:06:03.767115 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.767081 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-mcmvz" Sep 10 22:06:03.822485 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.822448 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2b50e35-7ca5-420b-a788-73f839011390-serving-cert\") pod \"insights-operator-5844b856db-hq6mn\" (UID: \"c2b50e35-7ca5-420b-a788-73f839011390\") " pod="openshift-insights/insights-operator-5844b856db-hq6mn" Sep 10 22:06:03.822618 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.822515 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2b50e35-7ca5-420b-a788-73f839011390-trusted-ca-bundle\") pod \"insights-operator-5844b856db-hq6mn\" (UID: \"c2b50e35-7ca5-420b-a788-73f839011390\") " pod="openshift-insights/insights-operator-5844b856db-hq6mn" Sep 10 22:06:03.822618 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.822552 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-765sx\" (UniqueName: \"kubernetes.io/projected/493e7732-8863-4f0f-b93e-bce47d8cf3de-kube-api-access-765sx\") pod \"network-check-target-wfb5h\" (UID: \"493e7732-8863-4f0f-b93e-bce47d8cf3de\") " pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:06:03.822618 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.822587 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6wv4\" (UniqueName: \"kubernetes.io/projected/0bc3a376-76d5-44b1-b9eb-0b988cd4b1dd-kube-api-access-g6wv4\") pod \"service-ca-operator-cf7c776b7-8ssql\" (UID: \"0bc3a376-76d5-44b1-b9eb-0b988cd4b1dd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8ssql" Sep 10 22:06:03.822830 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.822626 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c2b50e35-7ca5-420b-a788-73f839011390-snapshots\") pod \"insights-operator-5844b856db-hq6mn\" (UID: \"c2b50e35-7ca5-420b-a788-73f839011390\") " pod="openshift-insights/insights-operator-5844b856db-hq6mn" Sep 10 22:06:03.822830 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.822667 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2b50e35-7ca5-420b-a788-73f839011390-service-ca-bundle\") pod \"insights-operator-5844b856db-hq6mn\" (UID: \"c2b50e35-7ca5-420b-a788-73f839011390\") " pod="openshift-insights/insights-operator-5844b856db-hq6mn" Sep 10 22:06:03.822830 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.822698 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0bc3a376-76d5-44b1-b9eb-0b988cd4b1dd-serving-cert\") pod \"service-ca-operator-cf7c776b7-8ssql\" (UID: \"0bc3a376-76d5-44b1-b9eb-0b988cd4b1dd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8ssql" Sep 10 22:06:03.822830 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.822752 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0bc3a376-76d5-44b1-b9eb-0b988cd4b1dd-config\") pod \"service-ca-operator-cf7c776b7-8ssql\" (UID: \"0bc3a376-76d5-44b1-b9eb-0b988cd4b1dd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8ssql" Sep 10 22:06:03.822830 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.822781 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqms6\" (UniqueName: \"kubernetes.io/projected/c2b50e35-7ca5-420b-a788-73f839011390-kube-api-access-pqms6\") pod \"insights-operator-5844b856db-hq6mn\" (UID: \"c2b50e35-7ca5-420b-a788-73f839011390\") " pod="openshift-insights/insights-operator-5844b856db-hq6mn" Sep 10 22:06:03.822830 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.822811 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-5fl5b\" (UID: \"b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-5fl5b" Sep 10 22:06:03.823150 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.822844 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-5fl5b\" (UID: \"b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-5fl5b" Sep 10 22:06:03.823150 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.822872 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2b50e35-7ca5-420b-a788-73f839011390-tmp\") pod \"insights-operator-5844b856db-hq6mn\" (UID: \"c2b50e35-7ca5-420b-a788-73f839011390\") " pod="openshift-insights/insights-operator-5844b856db-hq6mn" Sep 10 22:06:03.824051 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.823425 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2b50e35-7ca5-420b-a788-73f839011390-tmp\") pod \"insights-operator-5844b856db-hq6mn\" (UID: \"c2b50e35-7ca5-420b-a788-73f839011390\") " pod="openshift-insights/insights-operator-5844b856db-hq6mn" Sep 10 22:06:03.824051 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.823520 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c2b50e35-7ca5-420b-a788-73f839011390-snapshots\") pod \"insights-operator-5844b856db-hq6mn\" (UID: \"c2b50e35-7ca5-420b-a788-73f839011390\") " pod="openshift-insights/insights-operator-5844b856db-hq6mn" Sep 10 22:06:03.824051 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.823722 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0bc3a376-76d5-44b1-b9eb-0b988cd4b1dd-config\") pod \"service-ca-operator-cf7c776b7-8ssql\" (UID: \"0bc3a376-76d5-44b1-b9eb-0b988cd4b1dd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8ssql" Sep 10 22:06:03.824051 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.823938 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2b50e35-7ca5-420b-a788-73f839011390-service-ca-bundle\") pod \"insights-operator-5844b856db-hq6mn\" (UID: \"c2b50e35-7ca5-420b-a788-73f839011390\") " pod="openshift-insights/insights-operator-5844b856db-hq6mn" Sep 10 22:06:03.824501 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.824476 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2b50e35-7ca5-420b-a788-73f839011390-trusted-ca-bundle\") pod \"insights-operator-5844b856db-hq6mn\" (UID: \"c2b50e35-7ca5-420b-a788-73f839011390\") " pod="openshift-insights/insights-operator-5844b856db-hq6mn" Sep 10 22:06:03.825315 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.825289 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2b50e35-7ca5-420b-a788-73f839011390-serving-cert\") pod \"insights-operator-5844b856db-hq6mn\" (UID: \"c2b50e35-7ca5-420b-a788-73f839011390\") " pod="openshift-insights/insights-operator-5844b856db-hq6mn" Sep 10 22:06:03.826232 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.826212 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0bc3a376-76d5-44b1-b9eb-0b988cd4b1dd-serving-cert\") pod \"service-ca-operator-cf7c776b7-8ssql\" (UID: \"0bc3a376-76d5-44b1-b9eb-0b988cd4b1dd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8ssql" Sep 10 22:06:03.827831 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.827804 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-765sx\" (UniqueName: \"kubernetes.io/projected/493e7732-8863-4f0f-b93e-bce47d8cf3de-kube-api-access-765sx\") pod \"network-check-target-wfb5h\" (UID: \"493e7732-8863-4f0f-b93e-bce47d8cf3de\") " pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:06:03.838322 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.838267 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6wv4\" (UniqueName: \"kubernetes.io/projected/0bc3a376-76d5-44b1-b9eb-0b988cd4b1dd-kube-api-access-g6wv4\") pod \"service-ca-operator-cf7c776b7-8ssql\" (UID: \"0bc3a376-76d5-44b1-b9eb-0b988cd4b1dd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8ssql" Sep 10 22:06:03.839526 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.839504 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqms6\" (UniqueName: \"kubernetes.io/projected/c2b50e35-7ca5-420b-a788-73f839011390-kube-api-access-pqms6\") pod \"insights-operator-5844b856db-hq6mn\" (UID: \"c2b50e35-7ca5-420b-a788-73f839011390\") " pod="openshift-insights/insights-operator-5844b856db-hq6mn" Sep 10 22:06:03.854961 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.854938 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8ssql" Sep 10 22:06:03.861749 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.861732 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-hq6mn" Sep 10 22:06:03.923572 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.923549 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-5fl5b\" (UID: \"b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-5fl5b" Sep 10 22:06:03.923662 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.923579 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-5fl5b\" (UID: \"b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-5fl5b" Sep 10 22:06:03.923832 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:03.923806 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 22:06:03.923933 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:03.923867 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50-networking-console-plugin-cert podName:b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:04.423851988 +0000 UTC m=+65.993341935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-5fl5b" (UID: "b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50") : secret "networking-console-plugin-cert" not found Sep 10 22:06:03.924201 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.924182 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-5fl5b\" (UID: \"b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-5fl5b" Sep 10 22:06:03.940727 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:03.940708 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m6nqn" Sep 10 22:06:04.036981 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:04.036937 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-mcmvz"] Sep 10 22:06:04.039366 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:06:04.039340 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9451128_97b3_4feb_b391_c2817898aa2c.slice/crio-36c6e5e36ba8fcb35a4cbd3873459e67c8896e5749d124008e6f4254eb758c6c WatchSource:0}: Error finding container 36c6e5e36ba8fcb35a4cbd3873459e67c8896e5749d124008e6f4254eb758c6c: Status 404 returned error can't find the container with id 36c6e5e36ba8fcb35a4cbd3873459e67c8896e5749d124008e6f4254eb758c6c Sep 10 22:06:04.051683 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:04.051663 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dmqhm\"" Sep 10 22:06:04.059165 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:04.059148 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:06:04.127257 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:04.127217 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-8ssql"] Sep 10 22:06:04.129700 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:06:04.129677 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc3a376_76d5_44b1_b9eb_0b988cd4b1dd.slice/crio-1e782239833bf7c8d9801f95972ffd31cf504b9bb61de76bb8dbcf3f59ac7465 WatchSource:0}: Error finding container 1e782239833bf7c8d9801f95972ffd31cf504b9bb61de76bb8dbcf3f59ac7465: Status 404 returned error can't find the container with id 1e782239833bf7c8d9801f95972ffd31cf504b9bb61de76bb8dbcf3f59ac7465 Sep 10 22:06:04.142044 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:04.142023 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-hq6mn"] Sep 10 22:06:04.144423 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:06:04.144402 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b50e35_7ca5_420b_a788_73f839011390.slice/crio-0e923764647a57a598e827a16ed5615552af1db8f59262eb7ea9bfc6fdbbcd34 WatchSource:0}: Error finding container 0e923764647a57a598e827a16ed5615552af1db8f59262eb7ea9bfc6fdbbcd34: Status 404 returned error can't find the container with id 0e923764647a57a598e827a16ed5615552af1db8f59262eb7ea9bfc6fdbbcd34 Sep 10 22:06:04.211421 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:04.211398 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m6nqn"] Sep 10 22:06:04.213862 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:06:04.213840 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod673499d4_211d_40e2_8a3f_95c74bc1d0f3.slice/crio-e252274b74798692b56d915f44518b58573febe1c0162549da62b368309c7c9e WatchSource:0}: Error finding container e252274b74798692b56d915f44518b58573febe1c0162549da62b368309c7c9e: Status 404 returned error can't find the container with id e252274b74798692b56d915f44518b58573febe1c0162549da62b368309c7c9e Sep 10 22:06:04.226360 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:04.226339 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1086cb8d-b834-4e42-87df-6db5b0bd9037-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fpgjg\" (UID: \"1086cb8d-b834-4e42-87df-6db5b0bd9037\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fpgjg" Sep 10 22:06:04.226449 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:04.226436 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 22:06:04.226493 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:04.226482 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1086cb8d-b834-4e42-87df-6db5b0bd9037-cluster-monitoring-operator-tls podName:1086cb8d-b834-4e42-87df-6db5b0bd9037 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:05.226468716 +0000 UTC m=+66.795958658 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1086cb8d-b834-4e42-87df-6db5b0bd9037-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fpgjg" (UID: "1086cb8d-b834-4e42-87df-6db5b0bd9037") : secret "cluster-monitoring-operator-tls" not found Sep 10 22:06:04.261681 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:04.261621 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-mcmvz" event={"ID":"e9451128-97b3-4feb-b391-c2817898aa2c","Type":"ContainerStarted","Data":"36c6e5e36ba8fcb35a4cbd3873459e67c8896e5749d124008e6f4254eb758c6c"} Sep 10 22:06:04.262731 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:04.262708 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8ssql" event={"ID":"0bc3a376-76d5-44b1-b9eb-0b988cd4b1dd","Type":"ContainerStarted","Data":"1e782239833bf7c8d9801f95972ffd31cf504b9bb61de76bb8dbcf3f59ac7465"} Sep 10 22:06:04.263787 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:04.263764 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m6nqn" event={"ID":"673499d4-211d-40e2-8a3f-95c74bc1d0f3","Type":"ContainerStarted","Data":"e252274b74798692b56d915f44518b58573febe1c0162549da62b368309c7c9e"} Sep 10 22:06:04.264769 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:04.264749 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-hq6mn" event={"ID":"c2b50e35-7ca5-420b-a788-73f839011390","Type":"ContainerStarted","Data":"0e923764647a57a598e827a16ed5615552af1db8f59262eb7ea9bfc6fdbbcd34"} Sep 10 22:06:04.334474 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:04.334451 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wfb5h"] Sep 10 22:06:04.337411 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:06:04.337385 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493e7732_8863_4f0f_b93e_bce47d8cf3de.slice/crio-371a49b656b3d883913af790b65f34b398a300e02e4ac43f6fa62776dce1114b WatchSource:0}: Error finding container 371a49b656b3d883913af790b65f34b398a300e02e4ac43f6fa62776dce1114b: Status 404 returned error can't find the container with id 371a49b656b3d883913af790b65f34b398a300e02e4ac43f6fa62776dce1114b Sep 10 22:06:04.427926 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:04.427885 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-5fl5b\" (UID: \"b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-5fl5b" Sep 10 22:06:04.428043 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:04.428025 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 22:06:04.428090 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:04.428084 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50-networking-console-plugin-cert podName:b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:05.428070059 +0000 UTC m=+66.997560001 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-5fl5b" (UID: "b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50") : secret "networking-console-plugin-cert" not found Sep 10 22:06:05.135917 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:05.135870 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:06:05.136386 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:05.135975 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0676c0bc-49e3-416a-97eb-70199abcf210-metrics-tls\") pod \"dns-default-lqltl\" (UID: \"0676c0bc-49e3-416a-97eb-70199abcf210\") " pod="openshift-dns/dns-default-lqltl" Sep 10 22:06:05.136386 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:05.136168 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 22:06:05.136386 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:05.136240 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0676c0bc-49e3-416a-97eb-70199abcf210-metrics-tls podName:0676c0bc-49e3-416a-97eb-70199abcf210 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:37.136212494 +0000 UTC m=+98.705702444 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0676c0bc-49e3-416a-97eb-70199abcf210-metrics-tls") pod "dns-default-lqltl" (UID: "0676c0bc-49e3-416a-97eb-70199abcf210") : secret "dns-default-metrics-tls" not found Sep 10 22:06:05.136766 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:05.136658 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 22:06:05.136766 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:05.136685 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bb7f54487-lqtc8: secret "image-registry-tls" not found Sep 10 22:06:05.136766 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:05.136734 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls podName:bd8bd0ce-a4ba-4134-89aa-224eace7e64d nodeName:}" failed. No retries permitted until 2026-09-10 22:06:37.136717135 +0000 UTC m=+98.706207095 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls") pod "image-registry-7bb7f54487-lqtc8" (UID: "bd8bd0ce-a4ba-4134-89aa-224eace7e64d") : secret "image-registry-tls" not found Sep 10 22:06:05.237189 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:05.237155 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1086cb8d-b834-4e42-87df-6db5b0bd9037-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fpgjg\" (UID: \"1086cb8d-b834-4e42-87df-6db5b0bd9037\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fpgjg" Sep 10 22:06:05.237359 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:05.237323 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 22:06:05.237423 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:05.237386 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1086cb8d-b834-4e42-87df-6db5b0bd9037-cluster-monitoring-operator-tls podName:1086cb8d-b834-4e42-87df-6db5b0bd9037 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:07.237368166 +0000 UTC m=+68.806858129 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1086cb8d-b834-4e42-87df-6db5b0bd9037-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fpgjg" (UID: "1086cb8d-b834-4e42-87df-6db5b0bd9037") : secret "cluster-monitoring-operator-tls" not found Sep 10 22:06:05.272421 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:05.272345 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wfb5h" event={"ID":"493e7732-8863-4f0f-b93e-bce47d8cf3de","Type":"ContainerStarted","Data":"371a49b656b3d883913af790b65f34b398a300e02e4ac43f6fa62776dce1114b"} Sep 10 22:06:05.439180 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:05.439095 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-5fl5b\" (UID: \"b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-5fl5b" Sep 10 22:06:05.439363 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:05.439350 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 22:06:05.439430 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:05.439417 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50-networking-console-plugin-cert podName:b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:07.439396847 +0000 UTC m=+69.008886807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-5fl5b" (UID: "b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50") : secret "networking-console-plugin-cert" not found Sep 10 22:06:07.261620 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:07.261467 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1086cb8d-b834-4e42-87df-6db5b0bd9037-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fpgjg\" (UID: \"1086cb8d-b834-4e42-87df-6db5b0bd9037\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fpgjg" Sep 10 22:06:07.262095 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:07.261631 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 22:06:07.262095 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:07.261717 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1086cb8d-b834-4e42-87df-6db5b0bd9037-cluster-monitoring-operator-tls podName:1086cb8d-b834-4e42-87df-6db5b0bd9037 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:11.261694876 +0000 UTC m=+72.831184840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1086cb8d-b834-4e42-87df-6db5b0bd9037-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fpgjg" (UID: "1086cb8d-b834-4e42-87df-6db5b0bd9037") : secret "cluster-monitoring-operator-tls" not found Sep 10 22:06:07.462942 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:07.462889 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-5fl5b\" (UID: \"b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-5fl5b" Sep 10 22:06:07.463088 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:07.463057 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 22:06:07.463131 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:07.463124 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50-networking-console-plugin-cert podName:b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:11.463107286 +0000 UTC m=+73.032597228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-5fl5b" (UID: "b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50") : secret "networking-console-plugin-cert" not found Sep 10 22:06:11.290042 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.290001 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1086cb8d-b834-4e42-87df-6db5b0bd9037-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fpgjg\" (UID: \"1086cb8d-b834-4e42-87df-6db5b0bd9037\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fpgjg" Sep 10 22:06:11.290565 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:11.290187 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 22:06:11.290565 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:11.290257 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1086cb8d-b834-4e42-87df-6db5b0bd9037-cluster-monitoring-operator-tls podName:1086cb8d-b834-4e42-87df-6db5b0bd9037 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:19.290236589 +0000 UTC m=+80.859726547 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1086cb8d-b834-4e42-87df-6db5b0bd9037-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fpgjg" (UID: "1086cb8d-b834-4e42-87df-6db5b0bd9037") : secret "cluster-monitoring-operator-tls" not found Sep 10 22:06:11.291148 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.291121 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-hq6mn" event={"ID":"c2b50e35-7ca5-420b-a788-73f839011390","Type":"ContainerStarted","Data":"1d6f7e02295e3472c138df9788a4ba3f41dc071d0bee8c386109f6ceb03f1caa"} Sep 10 22:06:11.292751 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.292729 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wfb5h" event={"ID":"493e7732-8863-4f0f-b93e-bce47d8cf3de","Type":"ContainerStarted","Data":"38c164dbb7cc19c19fcb5611a9b18ba4338011e091e1ce3e41b55a4a4dc02f54"} Sep 10 22:06:11.292858 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.292847 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:06:11.294387 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.294361 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-mcmvz" event={"ID":"e9451128-97b3-4feb-b391-c2817898aa2c","Type":"ContainerStarted","Data":"68e26c05a32fb6385eb4106aa2e529643fa6874ba5bf6d389a087d4e74eefc50"} Sep 10 22:06:11.295959 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.295934 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8ssql" event={"ID":"0bc3a376-76d5-44b1-b9eb-0b988cd4b1dd","Type":"ContainerStarted","Data":"c2679dfbdbc137f9e8d9acbc5834e596ba6e647d598a1602d64688fcccaf2988"} Sep 10 22:06:11.297474 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.297453 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m6nqn" event={"ID":"673499d4-211d-40e2-8a3f-95c74bc1d0f3","Type":"ContainerStarted","Data":"86153c1f61e13112eadc8bff76ea7639047f469f32b9359be3016e117a04fc4e"} Sep 10 22:06:11.309146 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.309102 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-hq6mn" podStartSLOduration=2.130224591 podStartE2EDuration="8.309088375s" totalImagesPullingTime="6.178863784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:06:11.307795138 +0000 UTC m=+72.877285104" watchObservedRunningTime="2026-09-10 22:06:11.309088375 +0000 UTC m=+72.878578331" Sep 10 22:06:11.323243 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.323190 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-m6nqn" podStartSLOduration=65.167074262 podStartE2EDuration="1m11.323170415s" totalImagesPullingTime="6.156096153s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:06:11.322513176 +0000 UTC m=+72.892003142" watchObservedRunningTime="2026-09-10 22:06:11.323170415 +0000 UTC m=+72.892660375" Sep 10 22:06:11.341374 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.340799 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-mcmvz" podStartSLOduration=2.024715732 podStartE2EDuration="8.340780564s" totalImagesPullingTime="6.316064832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:06:11.340012191 +0000 UTC m=+72.909502157" watchObservedRunningTime="2026-09-10 22:06:11.340780564 +0000 UTC m=+72.910270530" Sep 10 22:06:11.360173 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.360102 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wfb5h" podStartSLOduration=66.369172242 podStartE2EDuration="1m12.360082204s" totalImagesPullingTime="5.990909962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:04:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:06:11.359162919 +0000 UTC m=+72.928652885" watchObservedRunningTime="2026-09-10 22:06:11.360082204 +0000 UTC m=+72.929572166" Sep 10 22:06:11.380728 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.380673 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8ssql" podStartSLOduration=2.19045454 podStartE2EDuration="8.380650756s" totalImagesPullingTime="6.190196216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:06:11.378612141 +0000 UTC m=+72.948102104" watchObservedRunningTime="2026-09-10 22:06:11.380650756 +0000 UTC m=+72.950140721" Sep 10 22:06:11.445768 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.445736 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7bs46"] Sep 10 22:06:11.449237 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.449203 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:11.452199 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.452172 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 22:06:11.452805 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.452779 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 22:06:11.452913 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.452834 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bjqmf\"" Sep 10 22:06:11.459743 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.459723 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7bs46"] Sep 10 22:06:11.492549 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.492116 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/57656f7b-5fcb-46ed-9b3d-63296c4650d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:11.492549 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.492182 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/57656f7b-5fcb-46ed-9b3d-63296c4650d8-crio-socket\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:11.492549 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.492260 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/57656f7b-5fcb-46ed-9b3d-63296c4650d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:11.492549 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.492300 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-5fl5b\" (UID: \"b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-5fl5b" Sep 10 22:06:11.492549 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.492458 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/57656f7b-5fcb-46ed-9b3d-63296c4650d8-data-volume\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:11.492549 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.492486 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pv8s\" (UniqueName: \"kubernetes.io/projected/57656f7b-5fcb-46ed-9b3d-63296c4650d8-kube-api-access-5pv8s\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:11.493007 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:11.492663 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 22:06:11.493007 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:11.492742 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50-networking-console-plugin-cert podName:b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:19.49271965 +0000 UTC m=+81.062209598 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-5fl5b" (UID: "b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50") : secret "networking-console-plugin-cert" not found Sep 10 22:06:11.593300 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.593221 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/57656f7b-5fcb-46ed-9b3d-63296c4650d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:11.593300 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.593265 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/57656f7b-5fcb-46ed-9b3d-63296c4650d8-crio-socket\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:11.593488 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.593321 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/57656f7b-5fcb-46ed-9b3d-63296c4650d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:11.593488 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:11.593348 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 22:06:11.593488 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.593357 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/57656f7b-5fcb-46ed-9b3d-63296c4650d8-data-volume\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:11.593488 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:11.593429 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/57656f7b-5fcb-46ed-9b3d-63296c4650d8-kube-rbac-proxy-cm podName:57656f7b-5fcb-46ed-9b3d-63296c4650d8 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:12.093409126 +0000 UTC m=+73.662899085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/57656f7b-5fcb-46ed-9b3d-63296c4650d8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7bs46" (UID: "57656f7b-5fcb-46ed-9b3d-63296c4650d8") : configmap "kube-rbac-proxy" not found Sep 10 22:06:11.593488 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.593432 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/57656f7b-5fcb-46ed-9b3d-63296c4650d8-crio-socket\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:11.593488 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:11.593470 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 22:06:11.593488 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.593475 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pv8s\" (UniqueName: \"kubernetes.io/projected/57656f7b-5fcb-46ed-9b3d-63296c4650d8-kube-api-access-5pv8s\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:11.593728 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:11.593511 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57656f7b-5fcb-46ed-9b3d-63296c4650d8-insights-runtime-extractor-tls podName:57656f7b-5fcb-46ed-9b3d-63296c4650d8 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:12.093499746 +0000 UTC m=+73.662989704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/57656f7b-5fcb-46ed-9b3d-63296c4650d8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bs46" (UID: "57656f7b-5fcb-46ed-9b3d-63296c4650d8") : secret "insights-runtime-extractor-tls" not found Sep 10 22:06:11.593728 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.593701 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/57656f7b-5fcb-46ed-9b3d-63296c4650d8-data-volume\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:11.603495 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:11.603467 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pv8s\" (UniqueName: \"kubernetes.io/projected/57656f7b-5fcb-46ed-9b3d-63296c4650d8-kube-api-access-5pv8s\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:12.097126 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:12.097089 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/57656f7b-5fcb-46ed-9b3d-63296c4650d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:12.097305 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:12.097199 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 22:06:12.097305 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:12.097263 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/57656f7b-5fcb-46ed-9b3d-63296c4650d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:12.097305 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:12.097273 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/57656f7b-5fcb-46ed-9b3d-63296c4650d8-kube-rbac-proxy-cm podName:57656f7b-5fcb-46ed-9b3d-63296c4650d8 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:13.097257641 +0000 UTC m=+74.666747597 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/57656f7b-5fcb-46ed-9b3d-63296c4650d8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7bs46" (UID: "57656f7b-5fcb-46ed-9b3d-63296c4650d8") : configmap "kube-rbac-proxy" not found Sep 10 22:06:12.097458 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:12.097391 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 22:06:12.097458 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:12.097434 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57656f7b-5fcb-46ed-9b3d-63296c4650d8-insights-runtime-extractor-tls podName:57656f7b-5fcb-46ed-9b3d-63296c4650d8 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:13.097424028 +0000 UTC m=+74.666913991 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/57656f7b-5fcb-46ed-9b3d-63296c4650d8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bs46" (UID: "57656f7b-5fcb-46ed-9b3d-63296c4650d8") : secret "insights-runtime-extractor-tls" not found Sep 10 22:06:13.105915 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:13.105870 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/57656f7b-5fcb-46ed-9b3d-63296c4650d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:13.106369 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:13.106003 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/57656f7b-5fcb-46ed-9b3d-63296c4650d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:13.106369 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:13.106026 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 22:06:13.106369 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:13.106099 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57656f7b-5fcb-46ed-9b3d-63296c4650d8-insights-runtime-extractor-tls podName:57656f7b-5fcb-46ed-9b3d-63296c4650d8 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:15.106082313 +0000 UTC m=+76.675572255 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/57656f7b-5fcb-46ed-9b3d-63296c4650d8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bs46" (UID: "57656f7b-5fcb-46ed-9b3d-63296c4650d8") : secret "insights-runtime-extractor-tls" not found Sep 10 22:06:13.106369 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:13.106112 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 22:06:13.106369 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:13.106165 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/57656f7b-5fcb-46ed-9b3d-63296c4650d8-kube-rbac-proxy-cm podName:57656f7b-5fcb-46ed-9b3d-63296c4650d8 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:15.106148176 +0000 UTC m=+76.675638139 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/57656f7b-5fcb-46ed-9b3d-63296c4650d8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7bs46" (UID: "57656f7b-5fcb-46ed-9b3d-63296c4650d8") : configmap "kube-rbac-proxy" not found Sep 10 22:06:14.395532 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:14.395503 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zgwrg_0638522d-7b41-42c0-a93c-61fe7e00935b/dns-node-resolver/0.log" Sep 10 22:06:14.995599 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:14.995569 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ph2dv_c7114320-6f0b-4cc6-85a8-309ddda645af/node-ca/0.log" Sep 10 22:06:15.121130 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:15.121102 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/57656f7b-5fcb-46ed-9b3d-63296c4650d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:15.121279 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:15.121162 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/57656f7b-5fcb-46ed-9b3d-63296c4650d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:15.121279 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:15.121222 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 22:06:15.121279 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:15.121256 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 22:06:15.121379 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:15.121287 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/57656f7b-5fcb-46ed-9b3d-63296c4650d8-kube-rbac-proxy-cm podName:57656f7b-5fcb-46ed-9b3d-63296c4650d8 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:19.121271869 +0000 UTC m=+80.690761829 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/57656f7b-5fcb-46ed-9b3d-63296c4650d8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7bs46" (UID: "57656f7b-5fcb-46ed-9b3d-63296c4650d8") : configmap "kube-rbac-proxy" not found Sep 10 22:06:15.121379 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:15.121302 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57656f7b-5fcb-46ed-9b3d-63296c4650d8-insights-runtime-extractor-tls podName:57656f7b-5fcb-46ed-9b3d-63296c4650d8 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:19.121296042 +0000 UTC m=+80.690785998 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/57656f7b-5fcb-46ed-9b3d-63296c4650d8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bs46" (UID: "57656f7b-5fcb-46ed-9b3d-63296c4650d8") : secret "insights-runtime-extractor-tls" not found Sep 10 22:06:15.995528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:15.995502 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mp766_643143f3-0560-433b-9fca-05ef72574ee2/serve-healthcheck-canary/0.log" Sep 10 22:06:19.153437 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:19.153396 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/57656f7b-5fcb-46ed-9b3d-63296c4650d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:19.153889 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:19.153489 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/57656f7b-5fcb-46ed-9b3d-63296c4650d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:19.153889 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:19.153531 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 22:06:19.153889 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:19.153606 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/57656f7b-5fcb-46ed-9b3d-63296c4650d8-kube-rbac-proxy-cm podName:57656f7b-5fcb-46ed-9b3d-63296c4650d8 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:27.153589195 +0000 UTC m=+88.723079150 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/57656f7b-5fcb-46ed-9b3d-63296c4650d8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7bs46" (UID: "57656f7b-5fcb-46ed-9b3d-63296c4650d8") : configmap "kube-rbac-proxy" not found Sep 10 22:06:19.153889 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:19.153630 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 22:06:19.153889 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:19.153690 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57656f7b-5fcb-46ed-9b3d-63296c4650d8-insights-runtime-extractor-tls podName:57656f7b-5fcb-46ed-9b3d-63296c4650d8 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:27.153674363 +0000 UTC m=+88.723164322 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/57656f7b-5fcb-46ed-9b3d-63296c4650d8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bs46" (UID: "57656f7b-5fcb-46ed-9b3d-63296c4650d8") : secret "insights-runtime-extractor-tls" not found Sep 10 22:06:19.355207 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:19.355171 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1086cb8d-b834-4e42-87df-6db5b0bd9037-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fpgjg\" (UID: \"1086cb8d-b834-4e42-87df-6db5b0bd9037\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fpgjg" Sep 10 22:06:19.355378 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:19.355315 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 22:06:19.355378 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:19.355375 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1086cb8d-b834-4e42-87df-6db5b0bd9037-cluster-monitoring-operator-tls podName:1086cb8d-b834-4e42-87df-6db5b0bd9037 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:35.355359564 +0000 UTC m=+96.924849517 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1086cb8d-b834-4e42-87df-6db5b0bd9037-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fpgjg" (UID: "1086cb8d-b834-4e42-87df-6db5b0bd9037") : secret "cluster-monitoring-operator-tls" not found Sep 10 22:06:19.556128 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:19.556041 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-5fl5b\" (UID: \"b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-5fl5b" Sep 10 22:06:19.556291 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:19.556156 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 22:06:19.556291 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:19.556217 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50-networking-console-plugin-cert podName:b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:35.556203244 +0000 UTC m=+97.125693187 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-5fl5b" (UID: "b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50") : secret "networking-console-plugin-cert" not found Sep 10 22:06:27.211430 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:27.211293 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/57656f7b-5fcb-46ed-9b3d-63296c4650d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:27.211430 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:27.211385 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/57656f7b-5fcb-46ed-9b3d-63296c4650d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:27.211853 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:27.211575 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 22:06:27.211853 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:06:27.211660 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57656f7b-5fcb-46ed-9b3d-63296c4650d8-insights-runtime-extractor-tls podName:57656f7b-5fcb-46ed-9b3d-63296c4650d8 nodeName:}" failed. No retries permitted until 2026-09-10 22:06:43.211638428 +0000 UTC m=+104.781128391 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/57656f7b-5fcb-46ed-9b3d-63296c4650d8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bs46" (UID: "57656f7b-5fcb-46ed-9b3d-63296c4650d8") : secret "insights-runtime-extractor-tls" not found Sep 10 22:06:27.211968 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:27.211954 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/57656f7b-5fcb-46ed-9b3d-63296c4650d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:35.370200 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:35.370168 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1086cb8d-b834-4e42-87df-6db5b0bd9037-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fpgjg\" (UID: \"1086cb8d-b834-4e42-87df-6db5b0bd9037\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fpgjg" Sep 10 22:06:35.372514 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:35.372480 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1086cb8d-b834-4e42-87df-6db5b0bd9037-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fpgjg\" (UID: \"1086cb8d-b834-4e42-87df-6db5b0bd9037\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fpgjg" Sep 10 22:06:35.572216 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:35.572173 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-5fl5b\" (UID: \"b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-5fl5b" Sep 10 22:06:35.574566 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:35.574542 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-5fl5b\" (UID: \"b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-5fl5b" Sep 10 22:06:35.643333 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:35.643265 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fpgjg" Sep 10 22:06:35.747001 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:35.746975 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-5fl5b" Sep 10 22:06:35.912944 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:35.912922 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-fpgjg"] Sep 10 22:06:35.915044 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:06:35.915019 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1086cb8d_b834_4e42_87df_6db5b0bd9037.slice/crio-a79eab0ba07cf0865faf7b22a522ff51c5f1412e9a75212c03cc46fb5f1d37f7 WatchSource:0}: Error finding container a79eab0ba07cf0865faf7b22a522ff51c5f1412e9a75212c03cc46fb5f1d37f7: Status 404 returned error can't find the container with id a79eab0ba07cf0865faf7b22a522ff51c5f1412e9a75212c03cc46fb5f1d37f7 Sep 10 22:06:36.011728 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:36.011700 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-5fl5b"] Sep 10 22:06:36.014512 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:06:36.014488 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb21f2cfa_98bc_4a1b_9f4a_747ef2a45c50.slice/crio-bb13993e473614ecf4129cab1e9750628757db54f2b62d2ac6e2cf7829e36252 WatchSource:0}: Error finding container bb13993e473614ecf4129cab1e9750628757db54f2b62d2ac6e2cf7829e36252: Status 404 returned error can't find the container with id bb13993e473614ecf4129cab1e9750628757db54f2b62d2ac6e2cf7829e36252 Sep 10 22:06:36.375257 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:36.375222 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-5fl5b" event={"ID":"b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50","Type":"ContainerStarted","Data":"bb13993e473614ecf4129cab1e9750628757db54f2b62d2ac6e2cf7829e36252"} Sep 10 22:06:36.376393 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:36.376371 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fpgjg" event={"ID":"1086cb8d-b834-4e42-87df-6db5b0bd9037","Type":"ContainerStarted","Data":"a79eab0ba07cf0865faf7b22a522ff51c5f1412e9a75212c03cc46fb5f1d37f7"} Sep 10 22:06:36.913709 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:36.913669 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-769dbc5cb7-dqnxw"] Sep 10 22:06:36.922306 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:36.921291 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769dbc5cb7-dqnxw" Sep 10 22:06:36.928380 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:36.927254 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 10 22:06:36.928380 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:36.927528 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 10 22:06:36.928380 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:36.927707 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 10 22:06:36.928380 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:36.927914 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 10 22:06:36.942477 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:36.942425 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-769dbc5cb7-dqnxw"] Sep 10 22:06:36.983280 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:36.983251 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgw7v\" (UniqueName: \"kubernetes.io/projected/54494c5d-d65c-4273-9c15-9701958048b8-kube-api-access-zgw7v\") pod \"klusterlet-addon-workmgr-769dbc5cb7-dqnxw\" (UID: \"54494c5d-d65c-4273-9c15-9701958048b8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769dbc5cb7-dqnxw" Sep 10 22:06:36.983415 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:36.983318 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54494c5d-d65c-4273-9c15-9701958048b8-tmp\") pod \"klusterlet-addon-workmgr-769dbc5cb7-dqnxw\" (UID: \"54494c5d-d65c-4273-9c15-9701958048b8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769dbc5cb7-dqnxw" Sep 10 22:06:36.983415 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:36.983350 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/54494c5d-d65c-4273-9c15-9701958048b8-klusterlet-config\") pod \"klusterlet-addon-workmgr-769dbc5cb7-dqnxw\" (UID: \"54494c5d-d65c-4273-9c15-9701958048b8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769dbc5cb7-dqnxw" Sep 10 22:06:37.013137 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.013104 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-zhmbg"] Sep 10 22:06:37.017722 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.017693 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-zhmbg" Sep 10 22:06:37.020312 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.020289 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-5mhdf\"" Sep 10 22:06:37.020927 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.020894 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 10 22:06:37.021090 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.021007 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 10 22:06:37.032068 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.032042 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-zhmbg"] Sep 10 22:06:37.084104 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.084077 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr6wk\" (UniqueName: \"kubernetes.io/projected/4350b3b2-bbff-447f-b23a-124cc24aebef-kube-api-access-mr6wk\") pod \"downloads-76f99584f4-zhmbg\" (UID: \"4350b3b2-bbff-447f-b23a-124cc24aebef\") " pod="openshift-console/downloads-76f99584f4-zhmbg" Sep 10 22:06:37.084286 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.084125 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgw7v\" (UniqueName: \"kubernetes.io/projected/54494c5d-d65c-4273-9c15-9701958048b8-kube-api-access-zgw7v\") pod \"klusterlet-addon-workmgr-769dbc5cb7-dqnxw\" (UID: \"54494c5d-d65c-4273-9c15-9701958048b8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769dbc5cb7-dqnxw" Sep 10 22:06:37.084286 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.084225 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54494c5d-d65c-4273-9c15-9701958048b8-tmp\") pod \"klusterlet-addon-workmgr-769dbc5cb7-dqnxw\" (UID: \"54494c5d-d65c-4273-9c15-9701958048b8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769dbc5cb7-dqnxw" Sep 10 22:06:37.084286 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.084270 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/54494c5d-d65c-4273-9c15-9701958048b8-klusterlet-config\") pod \"klusterlet-addon-workmgr-769dbc5cb7-dqnxw\" (UID: \"54494c5d-d65c-4273-9c15-9701958048b8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769dbc5cb7-dqnxw" Sep 10 22:06:37.084671 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.084627 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54494c5d-d65c-4273-9c15-9701958048b8-tmp\") pod \"klusterlet-addon-workmgr-769dbc5cb7-dqnxw\" (UID: \"54494c5d-d65c-4273-9c15-9701958048b8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769dbc5cb7-dqnxw" Sep 10 22:06:37.087350 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.087324 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/54494c5d-d65c-4273-9c15-9701958048b8-klusterlet-config\") pod \"klusterlet-addon-workmgr-769dbc5cb7-dqnxw\" (UID: \"54494c5d-d65c-4273-9c15-9701958048b8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769dbc5cb7-dqnxw" Sep 10 22:06:37.103588 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.103533 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgw7v\" (UniqueName: \"kubernetes.io/projected/54494c5d-d65c-4273-9c15-9701958048b8-kube-api-access-zgw7v\") pod \"klusterlet-addon-workmgr-769dbc5cb7-dqnxw\" (UID: \"54494c5d-d65c-4273-9c15-9701958048b8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769dbc5cb7-dqnxw" Sep 10 22:06:37.185380 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.185300 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0676c0bc-49e3-416a-97eb-70199abcf210-metrics-tls\") pod \"dns-default-lqltl\" (UID: \"0676c0bc-49e3-416a-97eb-70199abcf210\") " pod="openshift-dns/dns-default-lqltl" Sep 10 22:06:37.185380 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.185379 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr6wk\" (UniqueName: \"kubernetes.io/projected/4350b3b2-bbff-447f-b23a-124cc24aebef-kube-api-access-mr6wk\") pod \"downloads-76f99584f4-zhmbg\" (UID: \"4350b3b2-bbff-447f-b23a-124cc24aebef\") " pod="openshift-console/downloads-76f99584f4-zhmbg" Sep 10 22:06:37.185597 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.185422 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:06:37.188286 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.188229 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0676c0bc-49e3-416a-97eb-70199abcf210-metrics-tls\") pod \"dns-default-lqltl\" (UID: \"0676c0bc-49e3-416a-97eb-70199abcf210\") " pod="openshift-dns/dns-default-lqltl" Sep 10 22:06:37.188410 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.188293 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls\") pod \"image-registry-7bb7f54487-lqtc8\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:06:37.195294 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.195246 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr6wk\" (UniqueName: \"kubernetes.io/projected/4350b3b2-bbff-447f-b23a-124cc24aebef-kube-api-access-mr6wk\") pod \"downloads-76f99584f4-zhmbg\" (UID: \"4350b3b2-bbff-447f-b23a-124cc24aebef\") " pod="openshift-console/downloads-76f99584f4-zhmbg" Sep 10 22:06:37.204088 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.204067 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mb5j2\"" Sep 10 22:06:37.212225 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.212194 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lqltl" Sep 10 22:06:37.241504 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.241477 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769dbc5cb7-dqnxw" Sep 10 22:06:37.328918 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.328865 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-zhmbg" Sep 10 22:06:37.490024 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.489945 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7vdxb\"" Sep 10 22:06:37.498084 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.498053 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:06:37.608767 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.608733 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lqltl"] Sep 10 22:06:37.611593 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.611563 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-769dbc5cb7-dqnxw"] Sep 10 22:06:37.615223 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:37.615200 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-zhmbg"] Sep 10 22:06:37.858408 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:06:37.858373 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4350b3b2_bbff_447f_b23a_124cc24aebef.slice/crio-2c03186d71aa7d9ec3928d4af3a915d6c8e44b851ddb79eaf49a4bc870d0fb0f WatchSource:0}: Error finding container 2c03186d71aa7d9ec3928d4af3a915d6c8e44b851ddb79eaf49a4bc870d0fb0f: Status 404 returned error can't find the container with id 2c03186d71aa7d9ec3928d4af3a915d6c8e44b851ddb79eaf49a4bc870d0fb0f Sep 10 22:06:37.858983 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:06:37.858956 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0676c0bc_49e3_416a_97eb_70199abcf210.slice/crio-9ba3c681ff635f8c20efbe316ce4e09f228c6aaa6dbe38c8ea4b5f032d82a6f3 WatchSource:0}: Error finding container 9ba3c681ff635f8c20efbe316ce4e09f228c6aaa6dbe38c8ea4b5f032d82a6f3: Status 404 returned error can't find the container with id 9ba3c681ff635f8c20efbe316ce4e09f228c6aaa6dbe38c8ea4b5f032d82a6f3 Sep 10 22:06:37.859789 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:06:37.859590 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54494c5d_d65c_4273_9c15_9701958048b8.slice/crio-e87b47f9d04eea4e2ef56420787db894f006a9f32d3132392999cd0bf72db771 WatchSource:0}: Error finding container e87b47f9d04eea4e2ef56420787db894f006a9f32d3132392999cd0bf72db771: Status 404 returned error can't find the container with id e87b47f9d04eea4e2ef56420787db894f006a9f32d3132392999cd0bf72db771 Sep 10 22:06:38.132781 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.132710 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bb7f54487-lqtc8"] Sep 10 22:06:38.135861 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:06:38.135830 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8bd0ce_a4ba_4134_89aa_224eace7e64d.slice/crio-2b17bbd3244759d37475fd39ff89b3c8cea7f7a71383573af1f3cf2475b7a3e1 WatchSource:0}: Error finding container 2b17bbd3244759d37475fd39ff89b3c8cea7f7a71383573af1f3cf2475b7a3e1: Status 404 returned error can't find the container with id 2b17bbd3244759d37475fd39ff89b3c8cea7f7a71383573af1f3cf2475b7a3e1 Sep 10 22:06:38.387228 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.387146 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-zhmbg" event={"ID":"4350b3b2-bbff-447f-b23a-124cc24aebef","Type":"ContainerStarted","Data":"2c03186d71aa7d9ec3928d4af3a915d6c8e44b851ddb79eaf49a4bc870d0fb0f"} Sep 10 22:06:38.389222 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.389165 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lqltl" event={"ID":"0676c0bc-49e3-416a-97eb-70199abcf210","Type":"ContainerStarted","Data":"9ba3c681ff635f8c20efbe316ce4e09f228c6aaa6dbe38c8ea4b5f032d82a6f3"} Sep 10 22:06:38.390566 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.390537 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769dbc5cb7-dqnxw" event={"ID":"54494c5d-d65c-4273-9c15-9701958048b8","Type":"ContainerStarted","Data":"e87b47f9d04eea4e2ef56420787db894f006a9f32d3132392999cd0bf72db771"} Sep 10 22:06:38.392293 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.392269 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" event={"ID":"bd8bd0ce-a4ba-4134-89aa-224eace7e64d","Type":"ContainerStarted","Data":"bbd831584d183fec4cf354e9392e668735aebf112669c4873ef41d8b01c622a1"} Sep 10 22:06:38.392388 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.392313 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" event={"ID":"bd8bd0ce-a4ba-4134-89aa-224eace7e64d","Type":"ContainerStarted","Data":"2b17bbd3244759d37475fd39ff89b3c8cea7f7a71383573af1f3cf2475b7a3e1"} Sep 10 22:06:38.392388 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.392376 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:06:38.393973 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.393934 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-5fl5b" event={"ID":"b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50","Type":"ContainerStarted","Data":"e5b170b39eed9a393fd0d84536c4828a4c810bb1aa77ac890c325607c3e15534"} Sep 10 22:06:38.395930 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.395882 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fpgjg" event={"ID":"1086cb8d-b834-4e42-87df-6db5b0bd9037","Type":"ContainerStarted","Data":"1f08cd58952b3dc1fc68871235a9d7d624983057e01f3a0488b0c90ab34808c4"} Sep 10 22:06:38.406603 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.406565 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lvr27"] Sep 10 22:06:38.409686 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.409667 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lvr27" Sep 10 22:06:38.413183 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.413157 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 10 22:06:38.413289 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.413162 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5plx5\"" Sep 10 22:06:38.419145 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.419098 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" podStartSLOduration=98.419079326 podStartE2EDuration="1m38.419079326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:06:38.415199426 +0000 UTC m=+99.984689394" watchObservedRunningTime="2026-09-10 22:06:38.419079326 +0000 UTC m=+99.988569292" Sep 10 22:06:38.420424 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.419403 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lvr27"] Sep 10 22:06:38.450165 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.450121 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fpgjg" podStartSLOduration=33.460649992 podStartE2EDuration="35.450107676s" totalImagesPullingTime="1.989457684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:06:38.450026649 +0000 UTC m=+100.019516614" watchObservedRunningTime="2026-09-10 22:06:38.450107676 +0000 UTC m=+100.019597642" Sep 10 22:06:38.464933 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.464833 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-5fl5b" podStartSLOduration=33.616336071 podStartE2EDuration="35.464816852s" totalImagesPullingTime="1.848480781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:06:38.464304214 +0000 UTC m=+100.033794176" watchObservedRunningTime="2026-09-10 22:06:38.464816852 +0000 UTC m=+100.034306819" Sep 10 22:06:38.500821 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.498756 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/40779462-2044-46c5-9c26-9dd79bd26267-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-lvr27\" (UID: \"40779462-2044-46c5-9c26-9dd79bd26267\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lvr27" Sep 10 22:06:38.560077 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.559616 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c66559948-ctjmt"] Sep 10 22:06:38.563367 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.563344 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:06:38.566931 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.566497 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 22:06:38.566931 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.566681 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 22:06:38.566931 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.566815 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 22:06:38.566931 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.566888 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6f7mc\"" Sep 10 22:06:38.567188 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.567131 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 22:06:38.567404 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.567386 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 22:06:38.574362 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.574336 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c66559948-ctjmt"] Sep 10 22:06:38.599735 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.599700 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/40779462-2044-46c5-9c26-9dd79bd26267-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-lvr27\" (UID: \"40779462-2044-46c5-9c26-9dd79bd26267\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lvr27" Sep 10 22:06:38.602834 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.602739 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/40779462-2044-46c5-9c26-9dd79bd26267-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-lvr27\" (UID: \"40779462-2044-46c5-9c26-9dd79bd26267\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lvr27" Sep 10 22:06:38.701032 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.700941 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8901fb7d-b8b1-4718-b530-bd480a55a012-oauth-serving-cert\") pod \"console-c66559948-ctjmt\" (UID: \"8901fb7d-b8b1-4718-b530-bd480a55a012\") " pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:06:38.701032 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.701000 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8901fb7d-b8b1-4718-b530-bd480a55a012-console-oauth-config\") pod \"console-c66559948-ctjmt\" (UID: \"8901fb7d-b8b1-4718-b530-bd480a55a012\") " pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:06:38.701235 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.701067 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-772ll\" (UniqueName: \"kubernetes.io/projected/8901fb7d-b8b1-4718-b530-bd480a55a012-kube-api-access-772ll\") pod \"console-c66559948-ctjmt\" (UID: \"8901fb7d-b8b1-4718-b530-bd480a55a012\") " pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:06:38.701235 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.701094 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8901fb7d-b8b1-4718-b530-bd480a55a012-service-ca\") pod \"console-c66559948-ctjmt\" (UID: \"8901fb7d-b8b1-4718-b530-bd480a55a012\") " pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:06:38.701235 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.701170 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8901fb7d-b8b1-4718-b530-bd480a55a012-console-serving-cert\") pod \"console-c66559948-ctjmt\" (UID: \"8901fb7d-b8b1-4718-b530-bd480a55a012\") " pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:06:38.701235 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.701207 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8901fb7d-b8b1-4718-b530-bd480a55a012-console-config\") pod \"console-c66559948-ctjmt\" (UID: \"8901fb7d-b8b1-4718-b530-bd480a55a012\") " pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:06:38.722125 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.721957 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lvr27" Sep 10 22:06:38.802694 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.802453 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8901fb7d-b8b1-4718-b530-bd480a55a012-console-oauth-config\") pod \"console-c66559948-ctjmt\" (UID: \"8901fb7d-b8b1-4718-b530-bd480a55a012\") " pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:06:38.802694 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.802520 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-772ll\" (UniqueName: \"kubernetes.io/projected/8901fb7d-b8b1-4718-b530-bd480a55a012-kube-api-access-772ll\") pod \"console-c66559948-ctjmt\" (UID: \"8901fb7d-b8b1-4718-b530-bd480a55a012\") " pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:06:38.802694 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.802545 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8901fb7d-b8b1-4718-b530-bd480a55a012-service-ca\") pod \"console-c66559948-ctjmt\" (UID: \"8901fb7d-b8b1-4718-b530-bd480a55a012\") " pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:06:38.802694 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.802614 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8901fb7d-b8b1-4718-b530-bd480a55a012-console-serving-cert\") pod \"console-c66559948-ctjmt\" (UID: \"8901fb7d-b8b1-4718-b530-bd480a55a012\") " pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:06:38.802694 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.802646 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8901fb7d-b8b1-4718-b530-bd480a55a012-console-config\") pod \"console-c66559948-ctjmt\" (UID: \"8901fb7d-b8b1-4718-b530-bd480a55a012\") " pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:06:38.802694 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.802688 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8901fb7d-b8b1-4718-b530-bd480a55a012-oauth-serving-cert\") pod \"console-c66559948-ctjmt\" (UID: \"8901fb7d-b8b1-4718-b530-bd480a55a012\") " pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:06:38.803482 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.803439 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8901fb7d-b8b1-4718-b530-bd480a55a012-oauth-serving-cert\") pod \"console-c66559948-ctjmt\" (UID: \"8901fb7d-b8b1-4718-b530-bd480a55a012\") " pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:06:38.804427 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.804404 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8901fb7d-b8b1-4718-b530-bd480a55a012-service-ca\") pod \"console-c66559948-ctjmt\" (UID: \"8901fb7d-b8b1-4718-b530-bd480a55a012\") " pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:06:38.805682 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.805586 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8901fb7d-b8b1-4718-b530-bd480a55a012-console-config\") pod \"console-c66559948-ctjmt\" (UID: \"8901fb7d-b8b1-4718-b530-bd480a55a012\") " pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:06:38.807054 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.807032 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8901fb7d-b8b1-4718-b530-bd480a55a012-console-oauth-config\") pod \"console-c66559948-ctjmt\" (UID: \"8901fb7d-b8b1-4718-b530-bd480a55a012\") " pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:06:38.811735 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.811693 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8901fb7d-b8b1-4718-b530-bd480a55a012-console-serving-cert\") pod \"console-c66559948-ctjmt\" (UID: \"8901fb7d-b8b1-4718-b530-bd480a55a012\") " pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:06:38.814914 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.814879 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-772ll\" (UniqueName: \"kubernetes.io/projected/8901fb7d-b8b1-4718-b530-bd480a55a012-kube-api-access-772ll\") pod \"console-c66559948-ctjmt\" (UID: \"8901fb7d-b8b1-4718-b530-bd480a55a012\") " pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:06:38.877618 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:38.877584 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:06:39.006726 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:39.005850 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lvr27"] Sep 10 22:06:39.010620 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:06:39.010585 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40779462_2044_46c5_9c26_9dd79bd26267.slice/crio-2093aea678455045f23b6bcd5bcaf849dde38adb7838963c6b764328e0381a07 WatchSource:0}: Error finding container 2093aea678455045f23b6bcd5bcaf849dde38adb7838963c6b764328e0381a07: Status 404 returned error can't find the container with id 2093aea678455045f23b6bcd5bcaf849dde38adb7838963c6b764328e0381a07 Sep 10 22:06:39.172363 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:39.172271 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c66559948-ctjmt"] Sep 10 22:06:39.182360 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:06:39.182318 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8901fb7d_b8b1_4718_b530_bd480a55a012.slice/crio-f18ba01ede7ea8b977fa6f35ab1df506e3309fbecb78c0e4e9507c0ce51a019b WatchSource:0}: Error finding container f18ba01ede7ea8b977fa6f35ab1df506e3309fbecb78c0e4e9507c0ce51a019b: Status 404 returned error can't find the container with id f18ba01ede7ea8b977fa6f35ab1df506e3309fbecb78c0e4e9507c0ce51a019b Sep 10 22:06:39.405442 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:39.405392 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lvr27" event={"ID":"40779462-2044-46c5-9c26-9dd79bd26267","Type":"ContainerStarted","Data":"2093aea678455045f23b6bcd5bcaf849dde38adb7838963c6b764328e0381a07"} Sep 10 22:06:39.407322 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:39.407279 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c66559948-ctjmt" event={"ID":"8901fb7d-b8b1-4718-b530-bd480a55a012","Type":"ContainerStarted","Data":"f18ba01ede7ea8b977fa6f35ab1df506e3309fbecb78c0e4e9507c0ce51a019b"} Sep 10 22:06:42.304750 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:42.304713 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wfb5h" Sep 10 22:06:43.242349 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:43.242315 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/57656f7b-5fcb-46ed-9b3d-63296c4650d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:43.245112 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:43.245090 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/57656f7b-5fcb-46ed-9b3d-63296c4650d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bs46\" (UID: \"57656f7b-5fcb-46ed-9b3d-63296c4650d8\") " pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:43.260017 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:43.259990 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7bs46" Sep 10 22:06:45.212421 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:45.212335 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7bs46"] Sep 10 22:06:45.226095 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:06:45.226063 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57656f7b_5fcb_46ed_9b3d_63296c4650d8.slice/crio-4010b104df5cc7dfd2c93a0128c2763c4de65605ff7ca2f920771682e2061e5f WatchSource:0}: Error finding container 4010b104df5cc7dfd2c93a0128c2763c4de65605ff7ca2f920771682e2061e5f: Status 404 returned error can't find the container with id 4010b104df5cc7dfd2c93a0128c2763c4de65605ff7ca2f920771682e2061e5f Sep 10 22:06:45.434794 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:45.434754 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lvr27" event={"ID":"40779462-2044-46c5-9c26-9dd79bd26267","Type":"ContainerStarted","Data":"cd786799927b3e98abf769182f9c614d399c1ac74904f6da3d8ea9a1c7405a47"} Sep 10 22:06:45.435224 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:45.435194 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lvr27" Sep 10 22:06:45.436885 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:45.436862 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7bs46" event={"ID":"57656f7b-5fcb-46ed-9b3d-63296c4650d8","Type":"ContainerStarted","Data":"3f4d1e9a4ac2e7cf3478b3e813f643d7f9099d3998473a54d419c0051ab29d53"} Sep 10 22:06:45.437065 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:45.436936 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7bs46" event={"ID":"57656f7b-5fcb-46ed-9b3d-63296c4650d8","Type":"ContainerStarted","Data":"4010b104df5cc7dfd2c93a0128c2763c4de65605ff7ca2f920771682e2061e5f"} Sep 10 22:06:45.439351 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:45.439288 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c66559948-ctjmt" event={"ID":"8901fb7d-b8b1-4718-b530-bd480a55a012","Type":"ContainerStarted","Data":"a8f42fc84682310afc09712abc9fa2b2e65d195d6583b9da72f08140abc25394"} Sep 10 22:06:45.440940 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:45.440918 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lvr27" Sep 10 22:06:45.442150 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:45.442125 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lqltl" event={"ID":"0676c0bc-49e3-416a-97eb-70199abcf210","Type":"ContainerStarted","Data":"2d2cca5a00a12874dd2fb9d11ab4d61da9d6feeeebf4ec7589d564df1e8ea6b9"} Sep 10 22:06:45.442254 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:45.442174 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lqltl" event={"ID":"0676c0bc-49e3-416a-97eb-70199abcf210","Type":"ContainerStarted","Data":"ba0f47de554dce4f912bca2d81d0165f4fea05bb7fc583917cf74405b37d99b0"} Sep 10 22:06:45.442254 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:45.442204 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lqltl" Sep 10 22:06:45.443738 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:45.443717 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769dbc5cb7-dqnxw" event={"ID":"54494c5d-d65c-4273-9c15-9701958048b8","Type":"ContainerStarted","Data":"56e816283170927d522e5f499988abf35c1fc21fd40560e48cc9aad8a240ad7b"} Sep 10 22:06:45.443948 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:45.443920 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769dbc5cb7-dqnxw" Sep 10 22:06:45.445457 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:45.445437 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769dbc5cb7-dqnxw" Sep 10 22:06:45.452484 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:45.452446 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lvr27" podStartSLOduration=1.5272869980000001 podStartE2EDuration="7.452433984s" totalImagesPullingTime="5.925146986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:06:45.451127199 +0000 UTC m=+107.020617160" watchObservedRunningTime="2026-09-10 22:06:45.452433984 +0000 UTC m=+107.021923946" Sep 10 22:06:45.471670 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:45.471617 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769dbc5cb7-dqnxw" podStartSLOduration=2.395902241 podStartE2EDuration="9.471603105s" totalImagesPullingTime="7.075700864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:06:45.470703566 +0000 UTC m=+107.040193531" watchObservedRunningTime="2026-09-10 22:06:45.471603105 +0000 UTC m=+107.041093069" Sep 10 22:06:45.514000 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:45.513876 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lqltl" podStartSLOduration=65.473707477 podStartE2EDuration="1m12.513860406s" totalImagesPullingTime="7.040152929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:05:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:06:45.513557839 +0000 UTC m=+107.083047805" watchObservedRunningTime="2026-09-10 22:06:45.513860406 +0000 UTC m=+107.083350373" Sep 10 22:06:45.536872 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:45.536819 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c66559948-ctjmt" podStartSLOduration=1.779629802 podStartE2EDuration="7.536802653s" totalImagesPullingTime="5.757172851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:06:45.534997553 +0000 UTC m=+107.104487520" watchObservedRunningTime="2026-09-10 22:06:45.536802653 +0000 UTC m=+107.106292615" Sep 10 22:06:47.455016 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:47.454954 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7bs46" event={"ID":"57656f7b-5fcb-46ed-9b3d-63296c4650d8","Type":"ContainerStarted","Data":"ad6c0abba0c55c2163525f1272993b4e0345153c335b35c4135ef3b89aecbedd"} Sep 10 22:06:48.461678 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:48.461638 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7bs46" event={"ID":"57656f7b-5fcb-46ed-9b3d-63296c4650d8","Type":"ContainerStarted","Data":"286ba808c61ffbad279eee65efd31a9c4bc157715175d956ee722a5001ea7f91"} Sep 10 22:06:48.484886 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:48.484777 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7bs46" podStartSLOduration=35.153919384 podStartE2EDuration="37.484755159s" totalImagesPullingTime="2.330835775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:06:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:06:48.482435246 +0000 UTC m=+110.051925230" watchObservedRunningTime="2026-09-10 22:06:48.484755159 +0000 UTC m=+110.054245126" Sep 10 22:06:48.878204 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:48.878162 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:06:48.878414 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:48.878241 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:06:48.883771 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:48.883742 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:06:49.469940 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:49.469909 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:06:52.170027 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.169987 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-g575r"] Sep 10 22:06:52.173693 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.173661 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.180112 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.180087 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 22:06:52.181057 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.181032 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 22:06:52.181453 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.181034 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 22:06:52.181453 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.181375 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x7rt7\"" Sep 10 22:06:52.181573 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.181516 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 22:06:52.322468 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.322422 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d70dda5-31b5-4c42-852f-83030940374f-sys\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.322639 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.322491 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d70dda5-31b5-4c42-852f-83030940374f-metrics-client-ca\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.322639 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.322523 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2d70dda5-31b5-4c42-852f-83030940374f-node-exporter-accelerators-collector-config\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.322639 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.322554 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2d70dda5-31b5-4c42-852f-83030940374f-node-exporter-tls\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.322639 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.322582 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2d70dda5-31b5-4c42-852f-83030940374f-root\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.322639 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.322601 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2d70dda5-31b5-4c42-852f-83030940374f-node-exporter-textfile\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.322944 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.322703 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2d70dda5-31b5-4c42-852f-83030940374f-node-exporter-wtmp\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.322944 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.322758 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d70dda5-31b5-4c42-852f-83030940374f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.322944 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.322791 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnstc\" (UniqueName: \"kubernetes.io/projected/2d70dda5-31b5-4c42-852f-83030940374f-kube-api-access-qnstc\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.424295 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.424202 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d70dda5-31b5-4c42-852f-83030940374f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.424295 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.424257 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnstc\" (UniqueName: \"kubernetes.io/projected/2d70dda5-31b5-4c42-852f-83030940374f-kube-api-access-qnstc\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.424511 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.424326 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d70dda5-31b5-4c42-852f-83030940374f-sys\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.424511 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.424362 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d70dda5-31b5-4c42-852f-83030940374f-metrics-client-ca\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.424511 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.424386 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2d70dda5-31b5-4c42-852f-83030940374f-node-exporter-accelerators-collector-config\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.424511 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.424416 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2d70dda5-31b5-4c42-852f-83030940374f-node-exporter-tls\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.424511 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.424444 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2d70dda5-31b5-4c42-852f-83030940374f-root\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.424511 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.424465 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2d70dda5-31b5-4c42-852f-83030940374f-node-exporter-textfile\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.424511 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.424497 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2d70dda5-31b5-4c42-852f-83030940374f-node-exporter-wtmp\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.424866 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.424668 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2d70dda5-31b5-4c42-852f-83030940374f-node-exporter-wtmp\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.424866 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.424759 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2d70dda5-31b5-4c42-852f-83030940374f-root\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.424866 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.424767 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d70dda5-31b5-4c42-852f-83030940374f-sys\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.425082 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.425062 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2d70dda5-31b5-4c42-852f-83030940374f-node-exporter-textfile\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.425368 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.425315 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2d70dda5-31b5-4c42-852f-83030940374f-node-exporter-accelerators-collector-config\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.425368 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.425333 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d70dda5-31b5-4c42-852f-83030940374f-metrics-client-ca\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.427239 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.427183 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d70dda5-31b5-4c42-852f-83030940374f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.427363 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.427322 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2d70dda5-31b5-4c42-852f-83030940374f-node-exporter-tls\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.436490 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.436462 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnstc\" (UniqueName: \"kubernetes.io/projected/2d70dda5-31b5-4c42-852f-83030940374f-kube-api-access-qnstc\") pod \"node-exporter-g575r\" (UID: \"2d70dda5-31b5-4c42-852f-83030940374f\") " pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:52.484461 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:52.484414 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g575r" Sep 10 22:06:53.189929 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.189779 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 22:06:53.194042 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.194012 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.199363 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.199334 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 10 22:06:53.199538 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.199443 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-sc78p\"" Sep 10 22:06:53.199621 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.199547 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 10 22:06:53.199621 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.199612 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 10 22:06:53.199709 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.199661 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 10 22:06:53.199797 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.199777 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 10 22:06:53.200445 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.200415 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 10 22:06:53.200930 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.200892 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 10 22:06:53.201028 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.200916 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 10 22:06:53.201334 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.201315 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 10 22:06:53.214081 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.214048 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 22:06:53.334010 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.333976 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.334010 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.334026 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-config-volume\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.334276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.334050 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.334276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.334080 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.334276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.334123 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f91df0a4-5135-4469-a643-aed828774d7d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.334276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.334198 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f91df0a4-5135-4469-a643-aed828774d7d-config-out\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.334276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.334272 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.334530 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.334312 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.334530 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.334337 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdwn9\" (UniqueName: \"kubernetes.io/projected/f91df0a4-5135-4469-a643-aed828774d7d-kube-api-access-kdwn9\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.334530 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.334389 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-web-config\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.334530 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.334410 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f91df0a4-5135-4469-a643-aed828774d7d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.334530 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.334468 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f91df0a4-5135-4469-a643-aed828774d7d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.334777 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.334543 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f91df0a4-5135-4469-a643-aed828774d7d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.435378 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.435337 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.435560 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.435396 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-config-volume\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.435560 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.435423 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.435691 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.435595 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.435691 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.435644 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f91df0a4-5135-4469-a643-aed828774d7d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.435691 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.435678 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f91df0a4-5135-4469-a643-aed828774d7d-config-out\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.435824 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.435728 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.435824 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.435780 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.435824 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.435803 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdwn9\" (UniqueName: \"kubernetes.io/projected/f91df0a4-5135-4469-a643-aed828774d7d-kube-api-access-kdwn9\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.436026 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.435845 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-web-config\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.436026 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.435868 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f91df0a4-5135-4469-a643-aed828774d7d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.436026 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.435896 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f91df0a4-5135-4469-a643-aed828774d7d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.436026 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.435976 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f91df0a4-5135-4469-a643-aed828774d7d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.436234 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.436170 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f91df0a4-5135-4469-a643-aed828774d7d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.437726 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.436663 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f91df0a4-5135-4469-a643-aed828774d7d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.437726 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.437403 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f91df0a4-5135-4469-a643-aed828774d7d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.438673 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.438628 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-config-volume\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.439206 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.439182 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.440196 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.440132 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.440589 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.440549 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f91df0a4-5135-4469-a643-aed828774d7d-config-out\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.440683 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.440596 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-web-config\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.440817 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.440794 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.441113 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.441094 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.441113 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.441103 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f91df0a4-5135-4469-a643-aed828774d7d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.442782 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.442760 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.447935 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.447875 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdwn9\" (UniqueName: \"kubernetes.io/projected/f91df0a4-5135-4469-a643-aed828774d7d-kube-api-access-kdwn9\") pod \"alertmanager-main-0\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.506669 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.506627 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:06:53.981415 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.981381 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5fdd7844f8-89nmk"] Sep 10 22:06:53.989995 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.989967 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:53.993982 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.993953 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 10 22:06:53.994117 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.993998 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 10 22:06:53.994252 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.994227 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 10 22:06:53.994341 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.994283 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-ql24s\"" Sep 10 22:06:53.994341 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.994285 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 10 22:06:53.995367 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.995344 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 10 22:06:53.995472 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:53.995377 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5rvpd312etqme\"" Sep 10 22:06:54.001546 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.001526 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5fdd7844f8-89nmk"] Sep 10 22:06:54.141794 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.141747 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/24cec8b0-92f4-4f81-be7b-7ca9df75bdf8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5fdd7844f8-89nmk\" (UID: \"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8\") " pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:54.142010 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.141826 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24cec8b0-92f4-4f81-be7b-7ca9df75bdf8-metrics-client-ca\") pod \"thanos-querier-5fdd7844f8-89nmk\" (UID: \"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8\") " pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:54.142010 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.141852 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfkmk\" (UniqueName: \"kubernetes.io/projected/24cec8b0-92f4-4f81-be7b-7ca9df75bdf8-kube-api-access-nfkmk\") pod \"thanos-querier-5fdd7844f8-89nmk\" (UID: \"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8\") " pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:54.142010 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.141889 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/24cec8b0-92f4-4f81-be7b-7ca9df75bdf8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5fdd7844f8-89nmk\" (UID: \"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8\") " pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:54.142010 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.141977 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/24cec8b0-92f4-4f81-be7b-7ca9df75bdf8-secret-thanos-querier-tls\") pod \"thanos-querier-5fdd7844f8-89nmk\" (UID: \"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8\") " pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:54.142211 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.142046 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/24cec8b0-92f4-4f81-be7b-7ca9df75bdf8-secret-grpc-tls\") pod \"thanos-querier-5fdd7844f8-89nmk\" (UID: \"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8\") " pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:54.142211 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.142105 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/24cec8b0-92f4-4f81-be7b-7ca9df75bdf8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5fdd7844f8-89nmk\" (UID: \"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8\") " pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:54.142211 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.142160 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/24cec8b0-92f4-4f81-be7b-7ca9df75bdf8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5fdd7844f8-89nmk\" (UID: \"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8\") " pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:54.243665 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.243570 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/24cec8b0-92f4-4f81-be7b-7ca9df75bdf8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5fdd7844f8-89nmk\" (UID: \"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8\") " pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:54.243665 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.243637 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/24cec8b0-92f4-4f81-be7b-7ca9df75bdf8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5fdd7844f8-89nmk\" (UID: \"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8\") " pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:54.243665 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.243667 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/24cec8b0-92f4-4f81-be7b-7ca9df75bdf8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5fdd7844f8-89nmk\" (UID: \"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8\") " pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:54.244281 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.243704 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24cec8b0-92f4-4f81-be7b-7ca9df75bdf8-metrics-client-ca\") pod \"thanos-querier-5fdd7844f8-89nmk\" (UID: \"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8\") " pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:54.244281 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.243728 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfkmk\" (UniqueName: \"kubernetes.io/projected/24cec8b0-92f4-4f81-be7b-7ca9df75bdf8-kube-api-access-nfkmk\") pod \"thanos-querier-5fdd7844f8-89nmk\" (UID: \"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8\") " pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:54.244281 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.243760 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/24cec8b0-92f4-4f81-be7b-7ca9df75bdf8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5fdd7844f8-89nmk\" (UID: \"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8\") " pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:54.244281 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.243791 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/24cec8b0-92f4-4f81-be7b-7ca9df75bdf8-secret-thanos-querier-tls\") pod \"thanos-querier-5fdd7844f8-89nmk\" (UID: \"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8\") " pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:54.244281 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.243824 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/24cec8b0-92f4-4f81-be7b-7ca9df75bdf8-secret-grpc-tls\") pod \"thanos-querier-5fdd7844f8-89nmk\" (UID: \"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8\") " pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:54.244630 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.244605 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24cec8b0-92f4-4f81-be7b-7ca9df75bdf8-metrics-client-ca\") pod \"thanos-querier-5fdd7844f8-89nmk\" (UID: \"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8\") " pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:54.247235 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.247207 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/24cec8b0-92f4-4f81-be7b-7ca9df75bdf8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5fdd7844f8-89nmk\" (UID: \"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8\") " pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:54.247348 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.247334 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/24cec8b0-92f4-4f81-be7b-7ca9df75bdf8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5fdd7844f8-89nmk\" (UID: \"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8\") " pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:54.247913 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.247873 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/24cec8b0-92f4-4f81-be7b-7ca9df75bdf8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5fdd7844f8-89nmk\" (UID: \"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8\") " pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:54.247994 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.247949 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/24cec8b0-92f4-4f81-be7b-7ca9df75bdf8-secret-thanos-querier-tls\") pod \"thanos-querier-5fdd7844f8-89nmk\" (UID: \"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8\") " pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:54.248036 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.248009 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/24cec8b0-92f4-4f81-be7b-7ca9df75bdf8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5fdd7844f8-89nmk\" (UID: \"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8\") " pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:54.248163 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.248140 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/24cec8b0-92f4-4f81-be7b-7ca9df75bdf8-secret-grpc-tls\") pod \"thanos-querier-5fdd7844f8-89nmk\" (UID: \"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8\") " pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:54.252892 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.252867 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfkmk\" (UniqueName: \"kubernetes.io/projected/24cec8b0-92f4-4f81-be7b-7ca9df75bdf8-kube-api-access-nfkmk\") pod \"thanos-querier-5fdd7844f8-89nmk\" (UID: \"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8\") " pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:54.301635 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:54.301608 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:06:55.450954 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:55.450923 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lqltl" Sep 10 22:06:55.630601 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:06:55.630559 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d70dda5_31b5_4c42_852f_83030940374f.slice/crio-d2b39963a09c98ecb50d5854cbb01d659e20ec48b63b85c92b04c072cba4f2f9 WatchSource:0}: Error finding container d2b39963a09c98ecb50d5854cbb01d659e20ec48b63b85c92b04c072cba4f2f9: Status 404 returned error can't find the container with id d2b39963a09c98ecb50d5854cbb01d659e20ec48b63b85c92b04c072cba4f2f9 Sep 10 22:06:55.910334 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:55.910299 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5fdd7844f8-89nmk"] Sep 10 22:06:55.912957 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:06:55.912833 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24cec8b0_92f4_4f81_be7b_7ca9df75bdf8.slice/crio-9b63748d2bd20f6019dd8fa5a6fd03257e07f0f11444314e2fab4e24a949e501 WatchSource:0}: Error finding container 9b63748d2bd20f6019dd8fa5a6fd03257e07f0f11444314e2fab4e24a949e501: Status 404 returned error can't find the container with id 9b63748d2bd20f6019dd8fa5a6fd03257e07f0f11444314e2fab4e24a949e501 Sep 10 22:06:55.916517 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:55.916489 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 22:06:55.920059 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:06:55.920030 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf91df0a4_5135_4469_a643_aed828774d7d.slice/crio-b63f8e278a0782f29ac1442912d40706b91494b5f974e2274d74d995ede62b9c WatchSource:0}: Error finding container b63f8e278a0782f29ac1442912d40706b91494b5f974e2274d74d995ede62b9c: Status 404 returned error can't find the container with id b63f8e278a0782f29ac1442912d40706b91494b5f974e2274d74d995ede62b9c Sep 10 22:06:56.501568 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.498660 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-zhmbg" event={"ID":"4350b3b2-bbff-447f-b23a-124cc24aebef","Type":"ContainerStarted","Data":"88ca6acf6aca8e1d822a6bcdf760115123a6418589d9b139e934c5e30c67e8ba"} Sep 10 22:06:56.502236 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.501771 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-zhmbg" Sep 10 22:06:56.504330 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.504138 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g575r" event={"ID":"2d70dda5-31b5-4c42-852f-83030940374f","Type":"ContainerStarted","Data":"d2b39963a09c98ecb50d5854cbb01d659e20ec48b63b85c92b04c072cba4f2f9"} Sep 10 22:06:56.506372 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.506280 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f91df0a4-5135-4469-a643-aed828774d7d","Type":"ContainerStarted","Data":"b63f8e278a0782f29ac1442912d40706b91494b5f974e2274d74d995ede62b9c"} Sep 10 22:06:56.508412 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.508320 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" event={"ID":"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8","Type":"ContainerStarted","Data":"9b63748d2bd20f6019dd8fa5a6fd03257e07f0f11444314e2fab4e24a949e501"} Sep 10 22:06:56.518423 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.518369 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-zhmbg" Sep 10 22:06:56.522238 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.522184 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-zhmbg" podStartSLOduration=2.660221359 podStartE2EDuration="20.522165709s" totalImagesPullingTime="17.86194435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:06:56.519737639 +0000 UTC m=+118.089227602" watchObservedRunningTime="2026-09-10 22:06:56.522165709 +0000 UTC m=+118.091655676" Sep 10 22:06:56.572474 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.571503 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-748b5b89c4-pljql"] Sep 10 22:06:56.578519 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.578364 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:06:56.581748 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.581711 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-b48wj\"" Sep 10 22:06:56.582706 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.582134 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 10 22:06:56.582706 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.582484 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-16kj7ai951hf6\"" Sep 10 22:06:56.583081 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.583038 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 10 22:06:56.583595 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.583540 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 10 22:06:56.584081 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.583977 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 10 22:06:56.586384 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.586308 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-748b5b89c4-pljql"] Sep 10 22:06:56.668195 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.667394 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cdf04798-bfc9-437a-8ed0-e3d893748ed0-client-ca-bundle\") pod \"metrics-server-748b5b89c4-pljql\" (UID: \"cdf04798-bfc9-437a-8ed0-e3d893748ed0\") " pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:06:56.668195 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.667469 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdf04798-bfc9-437a-8ed0-e3d893748ed0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-748b5b89c4-pljql\" (UID: \"cdf04798-bfc9-437a-8ed0-e3d893748ed0\") " pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:06:56.668195 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.667509 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cdf04798-bfc9-437a-8ed0-e3d893748ed0-metrics-server-audit-profiles\") pod \"metrics-server-748b5b89c4-pljql\" (UID: \"cdf04798-bfc9-437a-8ed0-e3d893748ed0\") " pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:06:56.668195 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.667569 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cdf04798-bfc9-437a-8ed0-e3d893748ed0-secret-metrics-server-client-certs\") pod \"metrics-server-748b5b89c4-pljql\" (UID: \"cdf04798-bfc9-437a-8ed0-e3d893748ed0\") " pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:06:56.668195 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.667592 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cdf04798-bfc9-437a-8ed0-e3d893748ed0-secret-metrics-server-tls\") pod \"metrics-server-748b5b89c4-pljql\" (UID: \"cdf04798-bfc9-437a-8ed0-e3d893748ed0\") " pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:06:56.668195 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.667631 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78v5n\" (UniqueName: \"kubernetes.io/projected/cdf04798-bfc9-437a-8ed0-e3d893748ed0-kube-api-access-78v5n\") pod \"metrics-server-748b5b89c4-pljql\" (UID: \"cdf04798-bfc9-437a-8ed0-e3d893748ed0\") " pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:06:56.668195 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.667656 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cdf04798-bfc9-437a-8ed0-e3d893748ed0-audit-log\") pod \"metrics-server-748b5b89c4-pljql\" (UID: \"cdf04798-bfc9-437a-8ed0-e3d893748ed0\") " pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:06:56.686314 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.686262 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-k77xr"] Sep 10 22:06:56.693702 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.693674 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-k77xr" Sep 10 22:06:56.697209 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.697130 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-j4cxm\"" Sep 10 22:06:56.698019 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.697678 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 10 22:06:56.701281 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.701200 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-k77xr"] Sep 10 22:06:56.770348 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.769355 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cdf04798-bfc9-437a-8ed0-e3d893748ed0-secret-metrics-server-client-certs\") pod \"metrics-server-748b5b89c4-pljql\" (UID: \"cdf04798-bfc9-437a-8ed0-e3d893748ed0\") " pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:06:56.770348 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.769401 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cdf04798-bfc9-437a-8ed0-e3d893748ed0-secret-metrics-server-tls\") pod \"metrics-server-748b5b89c4-pljql\" (UID: \"cdf04798-bfc9-437a-8ed0-e3d893748ed0\") " pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:06:56.770348 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.769448 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78v5n\" (UniqueName: \"kubernetes.io/projected/cdf04798-bfc9-437a-8ed0-e3d893748ed0-kube-api-access-78v5n\") pod \"metrics-server-748b5b89c4-pljql\" (UID: \"cdf04798-bfc9-437a-8ed0-e3d893748ed0\") " pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:06:56.770348 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.770226 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cdf04798-bfc9-437a-8ed0-e3d893748ed0-audit-log\") pod \"metrics-server-748b5b89c4-pljql\" (UID: \"cdf04798-bfc9-437a-8ed0-e3d893748ed0\") " pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:06:56.770348 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.770337 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cdf04798-bfc9-437a-8ed0-e3d893748ed0-client-ca-bundle\") pod \"metrics-server-748b5b89c4-pljql\" (UID: \"cdf04798-bfc9-437a-8ed0-e3d893748ed0\") " pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:06:56.770697 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.770405 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdf04798-bfc9-437a-8ed0-e3d893748ed0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-748b5b89c4-pljql\" (UID: \"cdf04798-bfc9-437a-8ed0-e3d893748ed0\") " pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:06:56.770697 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.770447 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cdf04798-bfc9-437a-8ed0-e3d893748ed0-metrics-server-audit-profiles\") pod \"metrics-server-748b5b89c4-pljql\" (UID: \"cdf04798-bfc9-437a-8ed0-e3d893748ed0\") " pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:06:56.772090 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.772044 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cdf04798-bfc9-437a-8ed0-e3d893748ed0-audit-log\") pod \"metrics-server-748b5b89c4-pljql\" (UID: \"cdf04798-bfc9-437a-8ed0-e3d893748ed0\") " pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:06:56.772768 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.772712 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdf04798-bfc9-437a-8ed0-e3d893748ed0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-748b5b89c4-pljql\" (UID: \"cdf04798-bfc9-437a-8ed0-e3d893748ed0\") " pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:06:56.773068 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.773025 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cdf04798-bfc9-437a-8ed0-e3d893748ed0-metrics-server-audit-profiles\") pod \"metrics-server-748b5b89c4-pljql\" (UID: \"cdf04798-bfc9-437a-8ed0-e3d893748ed0\") " pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:06:56.774422 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.774374 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cdf04798-bfc9-437a-8ed0-e3d893748ed0-secret-metrics-server-tls\") pod \"metrics-server-748b5b89c4-pljql\" (UID: \"cdf04798-bfc9-437a-8ed0-e3d893748ed0\") " pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:06:56.778307 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.778281 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cdf04798-bfc9-437a-8ed0-e3d893748ed0-secret-metrics-server-client-certs\") pod \"metrics-server-748b5b89c4-pljql\" (UID: \"cdf04798-bfc9-437a-8ed0-e3d893748ed0\") " pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:06:56.779100 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.779035 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78v5n\" (UniqueName: \"kubernetes.io/projected/cdf04798-bfc9-437a-8ed0-e3d893748ed0-kube-api-access-78v5n\") pod \"metrics-server-748b5b89c4-pljql\" (UID: \"cdf04798-bfc9-437a-8ed0-e3d893748ed0\") " pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:06:56.779730 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.779688 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cdf04798-bfc9-437a-8ed0-e3d893748ed0-client-ca-bundle\") pod \"metrics-server-748b5b89c4-pljql\" (UID: \"cdf04798-bfc9-437a-8ed0-e3d893748ed0\") " pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:06:56.871599 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.871393 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d55f82c-3a6d-4393-824e-8197bb503f20-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-k77xr\" (UID: \"1d55f82c-3a6d-4393-824e-8197bb503f20\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-k77xr" Sep 10 22:06:56.871599 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.871435 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl947\" (UniqueName: \"kubernetes.io/projected/1d55f82c-3a6d-4393-824e-8197bb503f20-kube-api-access-bl947\") pod \"monitoring-plugin-5d85778b9d-k77xr\" (UID: \"1d55f82c-3a6d-4393-824e-8197bb503f20\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-k77xr" Sep 10 22:06:56.903175 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.901537 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:06:56.973000 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.972960 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d55f82c-3a6d-4393-824e-8197bb503f20-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-k77xr\" (UID: \"1d55f82c-3a6d-4393-824e-8197bb503f20\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-k77xr" Sep 10 22:06:56.973000 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.973012 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl947\" (UniqueName: \"kubernetes.io/projected/1d55f82c-3a6d-4393-824e-8197bb503f20-kube-api-access-bl947\") pod \"monitoring-plugin-5d85778b9d-k77xr\" (UID: \"1d55f82c-3a6d-4393-824e-8197bb503f20\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-k77xr" Sep 10 22:06:56.976629 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.976570 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d55f82c-3a6d-4393-824e-8197bb503f20-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-k77xr\" (UID: \"1d55f82c-3a6d-4393-824e-8197bb503f20\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-k77xr" Sep 10 22:06:56.983273 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:56.983217 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl947\" (UniqueName: \"kubernetes.io/projected/1d55f82c-3a6d-4393-824e-8197bb503f20-kube-api-access-bl947\") pod \"monitoring-plugin-5d85778b9d-k77xr\" (UID: \"1d55f82c-3a6d-4393-824e-8197bb503f20\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-k77xr" Sep 10 22:06:57.011400 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:57.011177 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-k77xr" Sep 10 22:06:57.231473 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:57.231414 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c66559948-ctjmt"] Sep 10 22:06:57.246281 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:57.246205 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-748b5b89c4-pljql"] Sep 10 22:06:57.514874 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:57.514825 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" event={"ID":"cdf04798-bfc9-437a-8ed0-e3d893748ed0","Type":"ContainerStarted","Data":"f945a1ece468430cba354e9bc0e565c62c1b80c9a6229a9bf8a48ce85a615d2e"} Sep 10 22:06:57.517081 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:57.517051 2564 generic.go:356] "Generic (PLEG): container finished" podID="2d70dda5-31b5-4c42-852f-83030940374f" containerID="8b2cea207a71bc1f41326765222baf57cb68c9d2d8f8af533daea4900da16de9" exitCode=0 Sep 10 22:06:57.517339 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:57.517283 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g575r" event={"ID":"2d70dda5-31b5-4c42-852f-83030940374f","Type":"ContainerDied","Data":"8b2cea207a71bc1f41326765222baf57cb68c9d2d8f8af533daea4900da16de9"} Sep 10 22:06:57.744527 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:57.744447 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-k77xr"] Sep 10 22:06:58.199877 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.199842 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 22:06:58.222439 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.222396 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 22:06:58.222605 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.222573 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.227462 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.225557 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 10 22:06:58.227462 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.226005 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 10 22:06:58.227462 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.226178 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 10 22:06:58.227462 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.226453 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 10 22:06:58.227462 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.226647 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 10 22:06:58.227462 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.226817 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 10 22:06:58.227462 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.227033 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 10 22:06:58.227462 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.227215 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9f4rs\"" Sep 10 22:06:58.228928 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.228257 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 10 22:06:58.228928 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.228469 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 10 22:06:58.228928 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.228543 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 10 22:06:58.228928 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.228642 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 10 22:06:58.228928 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.228760 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 10 22:06:58.228928 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.228878 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 10 22:06:58.230563 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.230490 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 10 22:06:58.231181 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.231069 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fv1dc3gjvdb89\"" Sep 10 22:06:58.388276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.388227 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.388443 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.388313 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.388443 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.388336 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f318b0f-0166-465c-924a-8f246120877e-config-out\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.388443 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.388360 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.388443 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.388382 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.388443 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.388407 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.388700 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.388491 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.388700 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.388587 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-config\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.388700 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.388614 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.388700 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.388646 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.388700 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.388669 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.388937 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.388704 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-web-config\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.388937 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.388736 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.388937 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.388764 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jslch\" (UniqueName: \"kubernetes.io/projected/1f318b0f-0166-465c-924a-8f246120877e-kube-api-access-jslch\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.388937 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.388786 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.388937 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.388803 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1f318b0f-0166-465c-924a-8f246120877e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.388937 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.388822 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.388937 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.388874 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.389274 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.388973 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.389274 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.389028 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f318b0f-0166-465c-924a-8f246120877e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.417439 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:06:58.417406 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d55f82c_3a6d_4393_824e_8197bb503f20.slice/crio-3b921984bd3b76a0f439f4c6017b3a1f47fee4552eca9eafac0dc90bb564343d WatchSource:0}: Error finding container 3b921984bd3b76a0f439f4c6017b3a1f47fee4552eca9eafac0dc90bb564343d: Status 404 returned error can't find the container with id 3b921984bd3b76a0f439f4c6017b3a1f47fee4552eca9eafac0dc90bb564343d Sep 10 22:06:58.490094 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.490062 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.490198 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.490111 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.490198 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.490136 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.490198 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.490167 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.490370 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.490207 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-config\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.490370 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.490232 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.490370 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.490260 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.490370 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.490282 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.490370 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.490315 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-web-config\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.490370 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.490348 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.490662 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.490373 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jslch\" (UniqueName: \"kubernetes.io/projected/1f318b0f-0166-465c-924a-8f246120877e-kube-api-access-jslch\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.490662 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.490401 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.490662 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.490429 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1f318b0f-0166-465c-924a-8f246120877e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.490662 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.490459 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.490662 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.490490 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.490662 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.490520 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.490662 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.490561 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f318b0f-0166-465c-924a-8f246120877e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.490662 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.490610 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.491066 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.490673 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.491066 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.490703 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f318b0f-0166-465c-924a-8f246120877e-config-out\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.492759 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.492055 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.494443 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.493251 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.494443 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.493595 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.494443 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.493819 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1f318b0f-0166-465c-924a-8f246120877e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.494443 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.494388 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.495550 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.495305 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.496070 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.496048 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.496555 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.496533 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.498836 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.498727 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.498939 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.498848 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.500943 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.500008 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.500943 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.500115 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-config\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.500943 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.500555 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.500943 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.500739 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f318b0f-0166-465c-924a-8f246120877e-config-out\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.502196 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.501002 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-web-config\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.502196 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.501139 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jslch\" (UniqueName: \"kubernetes.io/projected/1f318b0f-0166-465c-924a-8f246120877e-kube-api-access-jslch\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.502196 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.501376 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.503570 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.502729 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f318b0f-0166-465c-924a-8f246120877e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.506727 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.505230 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.506727 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.505564 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.523869 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.523837 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g575r" event={"ID":"2d70dda5-31b5-4c42-852f-83030940374f","Type":"ContainerStarted","Data":"6c0dcf2d8903d22e758faf541fe68461db5320db5e849517cbb0a61f26c81b52"} Sep 10 22:06:58.525431 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.525402 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-k77xr" event={"ID":"1d55f82c-3a6d-4393-824e-8197bb503f20","Type":"ContainerStarted","Data":"3b921984bd3b76a0f439f4c6017b3a1f47fee4552eca9eafac0dc90bb564343d"} Sep 10 22:06:58.539115 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.539064 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:06:58.858247 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:58.858225 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 22:06:58.862108 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:06:58.862078 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f318b0f_0166_465c_924a_8f246120877e.slice/crio-662d0bbdd17b1de097ad8ee089d674e1fa925fbc232b71f3ffd8449260d65624 WatchSource:0}: Error finding container 662d0bbdd17b1de097ad8ee089d674e1fa925fbc232b71f3ffd8449260d65624: Status 404 returned error can't find the container with id 662d0bbdd17b1de097ad8ee089d674e1fa925fbc232b71f3ffd8449260d65624 Sep 10 22:06:59.111580 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:59.111551 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bb7f54487-lqtc8"] Sep 10 22:06:59.117151 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:59.117127 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:06:59.534804 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:59.534764 2564 generic.go:356] "Generic (PLEG): container finished" podID="f91df0a4-5135-4469-a643-aed828774d7d" containerID="3564baae1b1752c1c4b0ceafc3afed66a5cfd8e504912dc3198d75c6cc21f958" exitCode=0 Sep 10 22:06:59.535485 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:59.535440 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f91df0a4-5135-4469-a643-aed828774d7d","Type":"ContainerDied","Data":"3564baae1b1752c1c4b0ceafc3afed66a5cfd8e504912dc3198d75c6cc21f958"} Sep 10 22:06:59.544756 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:59.544659 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" event={"ID":"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8","Type":"ContainerStarted","Data":"31c42c84831a3472aa3b1f13de9b3afc366bc3cdc35e63a765826d32d69c5f66"} Sep 10 22:06:59.544756 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:59.544730 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" event={"ID":"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8","Type":"ContainerStarted","Data":"3322e2caa1583f5603131680f444b55d47c930da398e43f860ed29794a9bfc3f"} Sep 10 22:06:59.544948 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:59.544772 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" event={"ID":"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8","Type":"ContainerStarted","Data":"20d7987058ad081facc462d7217e8b3324d7ebdd336797f2302e0d70d0ae248f"} Sep 10 22:06:59.550800 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:59.550575 2564 generic.go:356] "Generic (PLEG): container finished" podID="1f318b0f-0166-465c-924a-8f246120877e" containerID="5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd" exitCode=0 Sep 10 22:06:59.550800 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:59.550663 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f318b0f-0166-465c-924a-8f246120877e","Type":"ContainerDied","Data":"5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd"} Sep 10 22:06:59.550800 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:59.550727 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f318b0f-0166-465c-924a-8f246120877e","Type":"ContainerStarted","Data":"662d0bbdd17b1de097ad8ee089d674e1fa925fbc232b71f3ffd8449260d65624"} Sep 10 22:06:59.556006 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:59.555982 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g575r" event={"ID":"2d70dda5-31b5-4c42-852f-83030940374f","Type":"ContainerStarted","Data":"bcb3e25e33b9a3d789e7f419cd55783b4c04ef5f22f16fd924bc6beaa7127b38"} Sep 10 22:06:59.613313 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:06:59.613244 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-g575r" podStartSLOduration=6.307609621 podStartE2EDuration="7.613225329s" totalImagesPullingTime="1.205026907s" totalInitContainerRuntime="100.588801ms" isStatefulPod=true podCreationTimestamp="2026-09-10 22:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:06:59.612211943 +0000 UTC m=+121.181701908" watchObservedRunningTime="2026-09-10 22:06:59.613225329 +0000 UTC m=+121.182715293" Sep 10 22:07:01.572975 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:01.572431 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-k77xr" event={"ID":"1d55f82c-3a6d-4393-824e-8197bb503f20","Type":"ContainerStarted","Data":"7950927ec2c8d0b19cf3d3f65eb83c0338d37c915d517a3d1624d97cf5779148"} Sep 10 22:07:01.579212 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:01.573803 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-k77xr" Sep 10 22:07:01.580754 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:01.579573 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" event={"ID":"cdf04798-bfc9-437a-8ed0-e3d893748ed0","Type":"ContainerStarted","Data":"f92b6865e0f9a64196398cb06a21aaf233d0992673eede52a2639ce51b564b2e"} Sep 10 22:07:01.581442 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:01.581394 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-k77xr" Sep 10 22:07:01.621573 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:01.621506 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-k77xr" podStartSLOduration=2.8248986069999997 podStartE2EDuration="5.621482851s" totalImagesPullingTime="2.796584244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:06:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:07:01.598769261 +0000 UTC m=+123.168259228" watchObservedRunningTime="2026-09-10 22:07:01.621482851 +0000 UTC m=+123.190972818" Sep 10 22:07:01.647572 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:01.645727 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" podStartSLOduration=2.417550043 podStartE2EDuration="5.645701481s" totalImagesPullingTime="3.228151438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:06:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:07:01.643565559 +0000 UTC m=+123.213055523" watchObservedRunningTime="2026-09-10 22:07:01.645701481 +0000 UTC m=+123.215191447" Sep 10 22:07:02.590851 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:02.590816 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" event={"ID":"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8","Type":"ContainerStarted","Data":"a7d10d0481966f51dbacdfb9929d604c2300660dc12a73721c480cfb12915d76"} Sep 10 22:07:03.601148 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:03.600861 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" event={"ID":"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8","Type":"ContainerStarted","Data":"5b034cb0e4033dc43a27b09ae3ac495325f43035dbb5a1f9cd8611d719112bff"} Sep 10 22:07:03.601148 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:03.601042 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" event={"ID":"24cec8b0-92f4-4f81-be7b-7ca9df75bdf8","Type":"ContainerStarted","Data":"c9db436a0072b2fec3aeafef5172201ce8ec94310634f5d5d946defc5e201cfb"} Sep 10 22:07:03.601148 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:03.601095 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:07:04.608049 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:04.608008 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f91df0a4-5135-4469-a643-aed828774d7d","Type":"ContainerStarted","Data":"d52e493fcec5931f2283342c4d23a9f40f8d801ff4ef8e65a0312d754a87e556"} Sep 10 22:07:04.616490 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:04.616461 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" Sep 10 22:07:04.645948 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:04.645862 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5fdd7844f8-89nmk" podStartSLOduration=6.300538136 podStartE2EDuration="11.645838769s" totalImagesPullingTime="5.345300633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:06:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:07:03.637430355 +0000 UTC m=+125.206920325" watchObservedRunningTime="2026-09-10 22:07:04.645838769 +0000 UTC m=+126.215328735" Sep 10 22:07:05.615414 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:05.615376 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f91df0a4-5135-4469-a643-aed828774d7d","Type":"ContainerStarted","Data":"2ba4b556adf46ac19129bc9009150aba50091e4938c5887abe52b0f6878b5d3e"} Sep 10 22:07:05.615864 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:05.615530 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f91df0a4-5135-4469-a643-aed828774d7d","Type":"ContainerStarted","Data":"a3f9f7d4a650231a8d54fc111e6c0411f060ce7d74987f9c5a569c8316d4125d"} Sep 10 22:07:05.615864 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:05.615573 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f91df0a4-5135-4469-a643-aed828774d7d","Type":"ContainerStarted","Data":"510ec7b38b032fbb1a6f25f589c53d6573ff417359ac28b683414a4556260f24"} Sep 10 22:07:05.615864 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:05.615611 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f91df0a4-5135-4469-a643-aed828774d7d","Type":"ContainerStarted","Data":"5f6b0090548acfa23acd76ebaaa91990eae84e95bb72186d8d090deba55b7b4b"} Sep 10 22:07:05.615864 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:05.615643 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f91df0a4-5135-4469-a643-aed828774d7d","Type":"ContainerStarted","Data":"abb5d9bd545dc254fe02cf7543098c22d76d36fbd91b22db0198c520a927605b"} Sep 10 22:07:05.619044 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:05.619017 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f318b0f-0166-465c-924a-8f246120877e","Type":"ContainerStarted","Data":"be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00"} Sep 10 22:07:05.619175 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:05.619076 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f318b0f-0166-465c-924a-8f246120877e","Type":"ContainerStarted","Data":"2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68"} Sep 10 22:07:05.619175 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:05.619109 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f318b0f-0166-465c-924a-8f246120877e","Type":"ContainerStarted","Data":"489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63"} Sep 10 22:07:05.619175 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:05.619138 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f318b0f-0166-465c-924a-8f246120877e","Type":"ContainerStarted","Data":"ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7"} Sep 10 22:07:05.619175 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:05.619176 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f318b0f-0166-465c-924a-8f246120877e","Type":"ContainerStarted","Data":"8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00"} Sep 10 22:07:05.619381 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:05.619202 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f318b0f-0166-465c-924a-8f246120877e","Type":"ContainerStarted","Data":"c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5"} Sep 10 22:07:05.646938 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:05.646808 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.37795851 podStartE2EDuration="12.646787976s" totalImagesPullingTime="7.245235623s" totalInitContainerRuntime="23.593843ms" isStatefulPod=false podCreationTimestamp="2026-09-10 22:06:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:07:05.645576402 +0000 UTC m=+127.215066368" watchObservedRunningTime="2026-09-10 22:07:05.646787976 +0000 UTC m=+127.216277942" Sep 10 22:07:05.681523 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:05.681470 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.553567772 podStartE2EDuration="7.681453138s" totalImagesPullingTime="5.088335593s" totalInitContainerRuntime="39.549773ms" isStatefulPod=false podCreationTimestamp="2026-09-10 22:06:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:07:05.679126811 +0000 UTC m=+127.248616776" watchObservedRunningTime="2026-09-10 22:07:05.681453138 +0000 UTC m=+127.250943103" Sep 10 22:07:07.698791 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:07.698754 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs\") pod \"network-metrics-daemon-fp72v\" (UID: \"d748c035-3205-4b2f-8b7e-a229ff50e466\") " pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:07:07.701209 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:07.701186 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d748c035-3205-4b2f-8b7e-a229ff50e466-metrics-certs\") pod \"network-metrics-daemon-fp72v\" (UID: \"d748c035-3205-4b2f-8b7e-a229ff50e466\") " pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:07:07.946164 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:07.946131 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8m774\"" Sep 10 22:07:07.954290 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:07.954229 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp72v" Sep 10 22:07:08.260367 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:08.260313 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fp72v"] Sep 10 22:07:08.262568 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:07:08.262538 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd748c035_3205_4b2f_8b7e_a229ff50e466.slice/crio-5d593cac6a5c74c320641b4d890a284e2f15f943cb3f1e21ff9c17a5bd3d48b4 WatchSource:0}: Error finding container 5d593cac6a5c74c320641b4d890a284e2f15f943cb3f1e21ff9c17a5bd3d48b4: Status 404 returned error can't find the container with id 5d593cac6a5c74c320641b4d890a284e2f15f943cb3f1e21ff9c17a5bd3d48b4 Sep 10 22:07:08.539520 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:08.539447 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:07:08.634077 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:08.634038 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fp72v" event={"ID":"d748c035-3205-4b2f-8b7e-a229ff50e466","Type":"ContainerStarted","Data":"5d593cac6a5c74c320641b4d890a284e2f15f943cb3f1e21ff9c17a5bd3d48b4"} Sep 10 22:07:10.644710 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:10.644676 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fp72v" event={"ID":"d748c035-3205-4b2f-8b7e-a229ff50e466","Type":"ContainerStarted","Data":"954748ce4851f8a29222c70e99c41b5e2b90a6bfffb23ab8627f0e472e653768"} Sep 10 22:07:10.645199 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:10.644727 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fp72v" event={"ID":"d748c035-3205-4b2f-8b7e-a229ff50e466","Type":"ContainerStarted","Data":"e656ee93ab5e1f9be02b824e9c458ee284fdd5a3e9ce9ffb76f2dc8ffa52d2d4"} Sep 10 22:07:10.666821 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:10.666762 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fp72v" podStartSLOduration=130.128407851 podStartE2EDuration="2m11.666747407s" totalImagesPullingTime="1.538339556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:04:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:07:10.664851568 +0000 UTC m=+132.234341532" watchObservedRunningTime="2026-09-10 22:07:10.666747407 +0000 UTC m=+132.236237418" Sep 10 22:07:16.671547 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:16.671505 2564 generic.go:356] "Generic (PLEG): container finished" podID="0bc3a376-76d5-44b1-b9eb-0b988cd4b1dd" containerID="c2679dfbdbc137f9e8d9acbc5834e596ba6e647d598a1602d64688fcccaf2988" exitCode=0 Sep 10 22:07:16.671547 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:16.671554 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8ssql" event={"ID":"0bc3a376-76d5-44b1-b9eb-0b988cd4b1dd","Type":"ContainerDied","Data":"c2679dfbdbc137f9e8d9acbc5834e596ba6e647d598a1602d64688fcccaf2988"} Sep 10 22:07:16.672075 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:16.671920 2564 scope.go:117] "RemoveContainer" containerID="c2679dfbdbc137f9e8d9acbc5834e596ba6e647d598a1602d64688fcccaf2988" Sep 10 22:07:16.904010 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:16.903971 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:07:16.904224 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:16.904123 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:07:17.677486 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:17.677447 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-8ssql" event={"ID":"0bc3a376-76d5-44b1-b9eb-0b988cd4b1dd","Type":"ContainerStarted","Data":"00bd8f898355982faf052c89b43a5bb4b31c9fbde783a30bf0db81bb443fd681"} Sep 10 22:07:22.259580 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.259523 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-c66559948-ctjmt" podUID="8901fb7d-b8b1-4718-b530-bd480a55a012" containerName="console" containerID="cri-o://a8f42fc84682310afc09712abc9fa2b2e65d195d6583b9da72f08140abc25394" gracePeriod=15 Sep 10 22:07:22.526044 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.525991 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:07:22.635637 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.635607 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8901fb7d-b8b1-4718-b530-bd480a55a012-oauth-serving-cert\") pod \"8901fb7d-b8b1-4718-b530-bd480a55a012\" (UID: \"8901fb7d-b8b1-4718-b530-bd480a55a012\") " Sep 10 22:07:22.635783 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.635649 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8901fb7d-b8b1-4718-b530-bd480a55a012-console-oauth-config\") pod \"8901fb7d-b8b1-4718-b530-bd480a55a012\" (UID: \"8901fb7d-b8b1-4718-b530-bd480a55a012\") " Sep 10 22:07:22.635783 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.635671 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8901fb7d-b8b1-4718-b530-bd480a55a012-console-serving-cert\") pod \"8901fb7d-b8b1-4718-b530-bd480a55a012\" (UID: \"8901fb7d-b8b1-4718-b530-bd480a55a012\") " Sep 10 22:07:22.635783 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.635698 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8901fb7d-b8b1-4718-b530-bd480a55a012-console-config\") pod \"8901fb7d-b8b1-4718-b530-bd480a55a012\" (UID: \"8901fb7d-b8b1-4718-b530-bd480a55a012\") " Sep 10 22:07:22.635939 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.635803 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8901fb7d-b8b1-4718-b530-bd480a55a012-service-ca\") pod \"8901fb7d-b8b1-4718-b530-bd480a55a012\" (UID: \"8901fb7d-b8b1-4718-b530-bd480a55a012\") " Sep 10 22:07:22.635989 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.635933 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-772ll\" (UniqueName: \"kubernetes.io/projected/8901fb7d-b8b1-4718-b530-bd480a55a012-kube-api-access-772ll\") pod \"8901fb7d-b8b1-4718-b530-bd480a55a012\" (UID: \"8901fb7d-b8b1-4718-b530-bd480a55a012\") " Sep 10 22:07:22.636195 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.636168 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8901fb7d-b8b1-4718-b530-bd480a55a012-console-config" (OuterVolumeSpecName: "console-config") pod "8901fb7d-b8b1-4718-b530-bd480a55a012" (UID: "8901fb7d-b8b1-4718-b530-bd480a55a012"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:07:22.636195 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.636167 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8901fb7d-b8b1-4718-b530-bd480a55a012-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8901fb7d-b8b1-4718-b530-bd480a55a012" (UID: "8901fb7d-b8b1-4718-b530-bd480a55a012"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:07:22.636371 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.636202 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8901fb7d-b8b1-4718-b530-bd480a55a012-service-ca" (OuterVolumeSpecName: "service-ca") pod "8901fb7d-b8b1-4718-b530-bd480a55a012" (UID: "8901fb7d-b8b1-4718-b530-bd480a55a012"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:07:22.636371 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.636312 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8901fb7d-b8b1-4718-b530-bd480a55a012-console-config\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:07:22.636371 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.636325 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8901fb7d-b8b1-4718-b530-bd480a55a012-service-ca\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:07:22.636371 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.636333 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8901fb7d-b8b1-4718-b530-bd480a55a012-oauth-serving-cert\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:07:22.638043 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.638021 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8901fb7d-b8b1-4718-b530-bd480a55a012-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8901fb7d-b8b1-4718-b530-bd480a55a012" (UID: "8901fb7d-b8b1-4718-b530-bd480a55a012"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:07:22.638118 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.638063 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8901fb7d-b8b1-4718-b530-bd480a55a012-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8901fb7d-b8b1-4718-b530-bd480a55a012" (UID: "8901fb7d-b8b1-4718-b530-bd480a55a012"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:07:22.638199 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.638181 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8901fb7d-b8b1-4718-b530-bd480a55a012-kube-api-access-772ll" (OuterVolumeSpecName: "kube-api-access-772ll") pod "8901fb7d-b8b1-4718-b530-bd480a55a012" (UID: "8901fb7d-b8b1-4718-b530-bd480a55a012"). InnerVolumeSpecName "kube-api-access-772ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:07:22.698962 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.698936 2564 generic.go:356] "Generic (PLEG): container finished" podID="8901fb7d-b8b1-4718-b530-bd480a55a012" containerID="a8f42fc84682310afc09712abc9fa2b2e65d195d6583b9da72f08140abc25394" exitCode=0 Sep 10 22:07:22.699081 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.698981 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c66559948-ctjmt" Sep 10 22:07:22.699081 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.699024 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c66559948-ctjmt" event={"ID":"8901fb7d-b8b1-4718-b530-bd480a55a012","Type":"ContainerDied","Data":"a8f42fc84682310afc09712abc9fa2b2e65d195d6583b9da72f08140abc25394"} Sep 10 22:07:22.699195 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.699081 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c66559948-ctjmt" event={"ID":"8901fb7d-b8b1-4718-b530-bd480a55a012","Type":"ContainerDied","Data":"f18ba01ede7ea8b977fa6f35ab1df506e3309fbecb78c0e4e9507c0ce51a019b"} Sep 10 22:07:22.699195 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.699107 2564 scope.go:117] "RemoveContainer" containerID="a8f42fc84682310afc09712abc9fa2b2e65d195d6583b9da72f08140abc25394" Sep 10 22:07:22.717222 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.717202 2564 scope.go:117] "RemoveContainer" containerID="a8f42fc84682310afc09712abc9fa2b2e65d195d6583b9da72f08140abc25394" Sep 10 22:07:22.717508 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:07:22.717486 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8f42fc84682310afc09712abc9fa2b2e65d195d6583b9da72f08140abc25394\": container with ID starting with a8f42fc84682310afc09712abc9fa2b2e65d195d6583b9da72f08140abc25394 not found: ID does not exist" containerID="a8f42fc84682310afc09712abc9fa2b2e65d195d6583b9da72f08140abc25394" Sep 10 22:07:22.717588 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.717515 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8f42fc84682310afc09712abc9fa2b2e65d195d6583b9da72f08140abc25394"} err="failed to get container status \"a8f42fc84682310afc09712abc9fa2b2e65d195d6583b9da72f08140abc25394\": rpc error: code = NotFound desc = could not find container \"a8f42fc84682310afc09712abc9fa2b2e65d195d6583b9da72f08140abc25394\": container with ID starting with a8f42fc84682310afc09712abc9fa2b2e65d195d6583b9da72f08140abc25394 not found: ID does not exist" Sep 10 22:07:22.723088 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.723067 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c66559948-ctjmt"] Sep 10 22:07:22.725306 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.725286 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c66559948-ctjmt"] Sep 10 22:07:22.737182 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.737161 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-772ll\" (UniqueName: \"kubernetes.io/projected/8901fb7d-b8b1-4718-b530-bd480a55a012-kube-api-access-772ll\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:07:22.737282 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.737184 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8901fb7d-b8b1-4718-b530-bd480a55a012-console-oauth-config\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:07:22.737282 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.737199 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8901fb7d-b8b1-4718-b530-bd480a55a012-console-serving-cert\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:07:22.936950 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:22.936919 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8901fb7d-b8b1-4718-b530-bd480a55a012" path="/var/lib/kubelet/pods/8901fb7d-b8b1-4718-b530-bd480a55a012/volumes" Sep 10 22:07:24.136776 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.136728 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" podUID="bd8bd0ce-a4ba-4134-89aa-224eace7e64d" containerName="registry" containerID="cri-o://bbd831584d183fec4cf354e9392e668735aebf112669c4873ef41d8b01c622a1" gracePeriod=30 Sep 10 22:07:24.411944 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.411879 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:07:24.552654 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.552626 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-trusted-ca\") pod \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " Sep 10 22:07:24.552813 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.552660 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-ca-trust-extracted\") pod \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " Sep 10 22:07:24.552813 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.552701 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-certificates\") pod \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " Sep 10 22:07:24.552813 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.552754 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-image-registry-private-configuration\") pod \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " Sep 10 22:07:24.552813 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.552788 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4m68r\" (UniqueName: \"kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-kube-api-access-4m68r\") pod \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " Sep 10 22:07:24.553047 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.552857 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls\") pod \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " Sep 10 22:07:24.553047 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.552917 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-installation-pull-secrets\") pod \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " Sep 10 22:07:24.553150 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.553059 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-bound-sa-token\") pod \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\" (UID: \"bd8bd0ce-a4ba-4134-89aa-224eace7e64d\") " Sep 10 22:07:24.553211 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.553181 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "bd8bd0ce-a4ba-4134-89aa-224eace7e64d" (UID: "bd8bd0ce-a4ba-4134-89aa-224eace7e64d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:07:24.553425 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.553400 2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-certificates\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:07:24.553884 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.553857 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "bd8bd0ce-a4ba-4134-89aa-224eace7e64d" (UID: "bd8bd0ce-a4ba-4134-89aa-224eace7e64d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:07:24.555395 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.555356 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-kube-api-access-4m68r" (OuterVolumeSpecName: "kube-api-access-4m68r") pod "bd8bd0ce-a4ba-4134-89aa-224eace7e64d" (UID: "bd8bd0ce-a4ba-4134-89aa-224eace7e64d"). InnerVolumeSpecName "kube-api-access-4m68r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:07:24.555500 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.555442 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "bd8bd0ce-a4ba-4134-89aa-224eace7e64d" (UID: "bd8bd0ce-a4ba-4134-89aa-224eace7e64d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:07:24.555718 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.555682 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "bd8bd0ce-a4ba-4134-89aa-224eace7e64d" (UID: "bd8bd0ce-a4ba-4134-89aa-224eace7e64d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:07:24.555718 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.555692 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bd8bd0ce-a4ba-4134-89aa-224eace7e64d" (UID: "bd8bd0ce-a4ba-4134-89aa-224eace7e64d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:07:24.556427 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.556404 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "bd8bd0ce-a4ba-4134-89aa-224eace7e64d" (UID: "bd8bd0ce-a4ba-4134-89aa-224eace7e64d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:07:24.561148 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.561120 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "bd8bd0ce-a4ba-4134-89aa-224eace7e64d" (UID: "bd8bd0ce-a4ba-4134-89aa-224eace7e64d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 22:07:24.654208 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.654183 2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-installation-pull-secrets\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:07:24.654208 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.654203 2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-bound-sa-token\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:07:24.654208 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.654213 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-trusted-ca\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:07:24.654379 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.654222 2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-ca-trust-extracted\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:07:24.654379 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.654231 2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-image-registry-private-configuration\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:07:24.654379 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.654241 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m68r\" (UniqueName: \"kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-kube-api-access-4m68r\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:07:24.654379 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.654249 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd8bd0ce-a4ba-4134-89aa-224eace7e64d-registry-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:07:24.709116 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.709050 2564 generic.go:356] "Generic (PLEG): container finished" podID="bd8bd0ce-a4ba-4134-89aa-224eace7e64d" containerID="bbd831584d183fec4cf354e9392e668735aebf112669c4873ef41d8b01c622a1" exitCode=0 Sep 10 22:07:24.709116 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.709097 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" Sep 10 22:07:24.709260 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.709130 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" event={"ID":"bd8bd0ce-a4ba-4134-89aa-224eace7e64d","Type":"ContainerDied","Data":"bbd831584d183fec4cf354e9392e668735aebf112669c4873ef41d8b01c622a1"} Sep 10 22:07:24.709260 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.709178 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bb7f54487-lqtc8" event={"ID":"bd8bd0ce-a4ba-4134-89aa-224eace7e64d","Type":"ContainerDied","Data":"2b17bbd3244759d37475fd39ff89b3c8cea7f7a71383573af1f3cf2475b7a3e1"} Sep 10 22:07:24.709260 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.709199 2564 scope.go:117] "RemoveContainer" containerID="bbd831584d183fec4cf354e9392e668735aebf112669c4873ef41d8b01c622a1" Sep 10 22:07:24.718349 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.718334 2564 scope.go:117] "RemoveContainer" containerID="bbd831584d183fec4cf354e9392e668735aebf112669c4873ef41d8b01c622a1" Sep 10 22:07:24.718644 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:07:24.718623 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbd831584d183fec4cf354e9392e668735aebf112669c4873ef41d8b01c622a1\": container with ID starting with bbd831584d183fec4cf354e9392e668735aebf112669c4873ef41d8b01c622a1 not found: ID does not exist" containerID="bbd831584d183fec4cf354e9392e668735aebf112669c4873ef41d8b01c622a1" Sep 10 22:07:24.718696 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.718653 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbd831584d183fec4cf354e9392e668735aebf112669c4873ef41d8b01c622a1"} err="failed to get container status \"bbd831584d183fec4cf354e9392e668735aebf112669c4873ef41d8b01c622a1\": rpc error: code = NotFound desc = could not find container \"bbd831584d183fec4cf354e9392e668735aebf112669c4873ef41d8b01c622a1\": container with ID starting with bbd831584d183fec4cf354e9392e668735aebf112669c4873ef41d8b01c622a1 not found: ID does not exist" Sep 10 22:07:24.732993 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.732967 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bb7f54487-lqtc8"] Sep 10 22:07:24.737312 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.737294 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7bb7f54487-lqtc8"] Sep 10 22:07:24.936206 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:24.936175 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd8bd0ce-a4ba-4134-89aa-224eace7e64d" path="/var/lib/kubelet/pods/bd8bd0ce-a4ba-4134-89aa-224eace7e64d/volumes" Sep 10 22:07:26.719449 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:26.719413 2564 generic.go:356] "Generic (PLEG): container finished" podID="c2b50e35-7ca5-420b-a788-73f839011390" containerID="1d6f7e02295e3472c138df9788a4ba3f41dc071d0bee8c386109f6ceb03f1caa" exitCode=0 Sep 10 22:07:26.719819 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:26.719490 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-hq6mn" event={"ID":"c2b50e35-7ca5-420b-a788-73f839011390","Type":"ContainerDied","Data":"1d6f7e02295e3472c138df9788a4ba3f41dc071d0bee8c386109f6ceb03f1caa"} Sep 10 22:07:26.719819 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:26.719813 2564 scope.go:117] "RemoveContainer" containerID="1d6f7e02295e3472c138df9788a4ba3f41dc071d0bee8c386109f6ceb03f1caa" Sep 10 22:07:27.726113 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:27.726073 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-hq6mn" event={"ID":"c2b50e35-7ca5-420b-a788-73f839011390","Type":"ContainerStarted","Data":"a3492e17958ed71f55e4e1f3786bc0ea98b0e4132380cd2f29efe7e238a1d39f"} Sep 10 22:07:36.908716 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:36.908685 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:07:36.912496 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:36.912476 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-748b5b89c4-pljql" Sep 10 22:07:58.539954 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:58.539840 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:07:58.558706 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:58.558676 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:07:58.868778 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:07:58.868746 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:12.307570 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:12.307523 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 22:08:12.308137 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:12.307857 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="alertmanager" containerID="cri-o://d52e493fcec5931f2283342c4d23a9f40f8d801ff4ef8e65a0312d754a87e556" gracePeriod=120 Sep 10 22:08:12.308137 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:12.307942 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="kube-rbac-proxy-web" containerID="cri-o://5f6b0090548acfa23acd76ebaaa91990eae84e95bb72186d8d090deba55b7b4b" gracePeriod=120 Sep 10 22:08:12.308137 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:12.307934 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="kube-rbac-proxy-metric" containerID="cri-o://a3f9f7d4a650231a8d54fc111e6c0411f060ce7d74987f9c5a569c8316d4125d" gracePeriod=120 Sep 10 22:08:12.308137 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:12.307960 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="kube-rbac-proxy" containerID="cri-o://510ec7b38b032fbb1a6f25f589c53d6573ff417359ac28b683414a4556260f24" gracePeriod=120 Sep 10 22:08:12.308137 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:12.307986 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="prom-label-proxy" containerID="cri-o://2ba4b556adf46ac19129bc9009150aba50091e4938c5887abe52b0f6878b5d3e" gracePeriod=120 Sep 10 22:08:12.308137 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:12.307960 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="config-reloader" containerID="cri-o://abb5d9bd545dc254fe02cf7543098c22d76d36fbd91b22db0198c520a927605b" gracePeriod=120 Sep 10 22:08:12.915143 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:12.915111 2564 generic.go:356] "Generic (PLEG): container finished" podID="f91df0a4-5135-4469-a643-aed828774d7d" containerID="2ba4b556adf46ac19129bc9009150aba50091e4938c5887abe52b0f6878b5d3e" exitCode=0 Sep 10 22:08:12.915143 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:12.915136 2564 generic.go:356] "Generic (PLEG): container finished" podID="f91df0a4-5135-4469-a643-aed828774d7d" containerID="a3f9f7d4a650231a8d54fc111e6c0411f060ce7d74987f9c5a569c8316d4125d" exitCode=0 Sep 10 22:08:12.915143 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:12.915143 2564 generic.go:356] "Generic (PLEG): container finished" podID="f91df0a4-5135-4469-a643-aed828774d7d" containerID="510ec7b38b032fbb1a6f25f589c53d6573ff417359ac28b683414a4556260f24" exitCode=0 Sep 10 22:08:12.915143 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:12.915148 2564 generic.go:356] "Generic (PLEG): container finished" podID="f91df0a4-5135-4469-a643-aed828774d7d" containerID="abb5d9bd545dc254fe02cf7543098c22d76d36fbd91b22db0198c520a927605b" exitCode=0 Sep 10 22:08:12.915400 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:12.915154 2564 generic.go:356] "Generic (PLEG): container finished" podID="f91df0a4-5135-4469-a643-aed828774d7d" containerID="d52e493fcec5931f2283342c4d23a9f40f8d801ff4ef8e65a0312d754a87e556" exitCode=0 Sep 10 22:08:12.915400 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:12.915184 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f91df0a4-5135-4469-a643-aed828774d7d","Type":"ContainerDied","Data":"2ba4b556adf46ac19129bc9009150aba50091e4938c5887abe52b0f6878b5d3e"} Sep 10 22:08:12.915400 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:12.915249 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f91df0a4-5135-4469-a643-aed828774d7d","Type":"ContainerDied","Data":"a3f9f7d4a650231a8d54fc111e6c0411f060ce7d74987f9c5a569c8316d4125d"} Sep 10 22:08:12.915400 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:12.915274 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f91df0a4-5135-4469-a643-aed828774d7d","Type":"ContainerDied","Data":"510ec7b38b032fbb1a6f25f589c53d6573ff417359ac28b683414a4556260f24"} Sep 10 22:08:12.915400 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:12.915298 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f91df0a4-5135-4469-a643-aed828774d7d","Type":"ContainerDied","Data":"abb5d9bd545dc254fe02cf7543098c22d76d36fbd91b22db0198c520a927605b"} Sep 10 22:08:12.915400 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:12.915320 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f91df0a4-5135-4469-a643-aed828774d7d","Type":"ContainerDied","Data":"d52e493fcec5931f2283342c4d23a9f40f8d801ff4ef8e65a0312d754a87e556"} Sep 10 22:08:13.548213 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.548177 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:13.668406 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.668298 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-cluster-tls-config\") pod \"f91df0a4-5135-4469-a643-aed828774d7d\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " Sep 10 22:08:13.668406 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.668338 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdwn9\" (UniqueName: \"kubernetes.io/projected/f91df0a4-5135-4469-a643-aed828774d7d-kube-api-access-kdwn9\") pod \"f91df0a4-5135-4469-a643-aed828774d7d\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " Sep 10 22:08:13.668406 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.668356 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f91df0a4-5135-4469-a643-aed828774d7d-tls-assets\") pod \"f91df0a4-5135-4469-a643-aed828774d7d\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " Sep 10 22:08:13.668406 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.668386 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-web-config\") pod \"f91df0a4-5135-4469-a643-aed828774d7d\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " Sep 10 22:08:13.668712 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.668577 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-secret-alertmanager-kube-rbac-proxy-web\") pod \"f91df0a4-5135-4469-a643-aed828774d7d\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " Sep 10 22:08:13.668712 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.668620 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f91df0a4-5135-4469-a643-aed828774d7d-config-out\") pod \"f91df0a4-5135-4469-a643-aed828774d7d\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " Sep 10 22:08:13.668712 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.668656 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f91df0a4-5135-4469-a643-aed828774d7d\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " Sep 10 22:08:13.668712 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.668686 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f91df0a4-5135-4469-a643-aed828774d7d-metrics-client-ca\") pod \"f91df0a4-5135-4469-a643-aed828774d7d\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " Sep 10 22:08:13.668846 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.668729 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-config-volume\") pod \"f91df0a4-5135-4469-a643-aed828774d7d\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " Sep 10 22:08:13.668846 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.668786 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-secret-alertmanager-main-tls\") pod \"f91df0a4-5135-4469-a643-aed828774d7d\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " Sep 10 22:08:13.668846 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.668813 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f91df0a4-5135-4469-a643-aed828774d7d-alertmanager-trusted-ca-bundle\") pod \"f91df0a4-5135-4469-a643-aed828774d7d\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " Sep 10 22:08:13.668846 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.668841 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f91df0a4-5135-4469-a643-aed828774d7d-alertmanager-main-db\") pod \"f91df0a4-5135-4469-a643-aed828774d7d\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " Sep 10 22:08:13.669062 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.668867 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-secret-alertmanager-kube-rbac-proxy\") pod \"f91df0a4-5135-4469-a643-aed828774d7d\" (UID: \"f91df0a4-5135-4469-a643-aed828774d7d\") " Sep 10 22:08:13.669526 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.669492 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f91df0a4-5135-4469-a643-aed828774d7d-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "f91df0a4-5135-4469-a643-aed828774d7d" (UID: "f91df0a4-5135-4469-a643-aed828774d7d"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:08:13.669722 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.669695 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f91df0a4-5135-4469-a643-aed828774d7d-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "f91df0a4-5135-4469-a643-aed828774d7d" (UID: "f91df0a4-5135-4469-a643-aed828774d7d"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:08:13.669919 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.669867 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91df0a4-5135-4469-a643-aed828774d7d-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "f91df0a4-5135-4469-a643-aed828774d7d" (UID: "f91df0a4-5135-4469-a643-aed828774d7d"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 22:08:13.671667 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.671631 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "f91df0a4-5135-4469-a643-aed828774d7d" (UID: "f91df0a4-5135-4469-a643-aed828774d7d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:08:13.672078 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.672040 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f91df0a4-5135-4469-a643-aed828774d7d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f91df0a4-5135-4469-a643-aed828774d7d" (UID: "f91df0a4-5135-4469-a643-aed828774d7d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:08:13.672078 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.672062 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f91df0a4-5135-4469-a643-aed828774d7d-kube-api-access-kdwn9" (OuterVolumeSpecName: "kube-api-access-kdwn9") pod "f91df0a4-5135-4469-a643-aed828774d7d" (UID: "f91df0a4-5135-4469-a643-aed828774d7d"). InnerVolumeSpecName "kube-api-access-kdwn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:08:13.672234 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.672180 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "f91df0a4-5135-4469-a643-aed828774d7d" (UID: "f91df0a4-5135-4469-a643-aed828774d7d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:08:13.672234 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.672197 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "f91df0a4-5135-4469-a643-aed828774d7d" (UID: "f91df0a4-5135-4469-a643-aed828774d7d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:08:13.672345 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.672279 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "f91df0a4-5135-4469-a643-aed828774d7d" (UID: "f91df0a4-5135-4469-a643-aed828774d7d"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:08:13.672777 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.672745 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91df0a4-5135-4469-a643-aed828774d7d-config-out" (OuterVolumeSpecName: "config-out") pod "f91df0a4-5135-4469-a643-aed828774d7d" (UID: "f91df0a4-5135-4469-a643-aed828774d7d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 22:08:13.673190 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.673165 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-config-volume" (OuterVolumeSpecName: "config-volume") pod "f91df0a4-5135-4469-a643-aed828774d7d" (UID: "f91df0a4-5135-4469-a643-aed828774d7d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:08:13.675854 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.675825 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "f91df0a4-5135-4469-a643-aed828774d7d" (UID: "f91df0a4-5135-4469-a643-aed828774d7d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:08:13.683056 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.683023 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-web-config" (OuterVolumeSpecName: "web-config") pod "f91df0a4-5135-4469-a643-aed828774d7d" (UID: "f91df0a4-5135-4469-a643-aed828774d7d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:08:13.770086 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.770038 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:13.770086 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.770081 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f91df0a4-5135-4469-a643-aed828774d7d-config-out\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:13.770086 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.770091 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:13.770086 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.770101 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f91df0a4-5135-4469-a643-aed828774d7d-metrics-client-ca\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:13.770345 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.770111 2564 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-config-volume\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:13.770345 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.770123 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-secret-alertmanager-main-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:13.770345 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.770131 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f91df0a4-5135-4469-a643-aed828774d7d-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:13.770345 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.770139 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f91df0a4-5135-4469-a643-aed828774d7d-alertmanager-main-db\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:13.770345 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.770147 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:13.770345 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.770156 2564 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-cluster-tls-config\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:13.770345 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.770166 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdwn9\" (UniqueName: \"kubernetes.io/projected/f91df0a4-5135-4469-a643-aed828774d7d-kube-api-access-kdwn9\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:13.770345 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.770175 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f91df0a4-5135-4469-a643-aed828774d7d-tls-assets\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:13.770345 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.770183 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f91df0a4-5135-4469-a643-aed828774d7d-web-config\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:13.922104 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.922018 2564 generic.go:356] "Generic (PLEG): container finished" podID="f91df0a4-5135-4469-a643-aed828774d7d" containerID="5f6b0090548acfa23acd76ebaaa91990eae84e95bb72186d8d090deba55b7b4b" exitCode=0 Sep 10 22:08:13.922242 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.922100 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f91df0a4-5135-4469-a643-aed828774d7d","Type":"ContainerDied","Data":"5f6b0090548acfa23acd76ebaaa91990eae84e95bb72186d8d090deba55b7b4b"} Sep 10 22:08:13.922242 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.922167 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f91df0a4-5135-4469-a643-aed828774d7d","Type":"ContainerDied","Data":"b63f8e278a0782f29ac1442912d40706b91494b5f974e2274d74d995ede62b9c"} Sep 10 22:08:13.922242 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.922105 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:13.922242 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.922205 2564 scope.go:117] "RemoveContainer" containerID="2ba4b556adf46ac19129bc9009150aba50091e4938c5887abe52b0f6878b5d3e" Sep 10 22:08:13.929818 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.929802 2564 scope.go:117] "RemoveContainer" containerID="a3f9f7d4a650231a8d54fc111e6c0411f060ce7d74987f9c5a569c8316d4125d" Sep 10 22:08:13.936454 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.936436 2564 scope.go:117] "RemoveContainer" containerID="510ec7b38b032fbb1a6f25f589c53d6573ff417359ac28b683414a4556260f24" Sep 10 22:08:13.943113 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.943097 2564 scope.go:117] "RemoveContainer" containerID="5f6b0090548acfa23acd76ebaaa91990eae84e95bb72186d8d090deba55b7b4b" Sep 10 22:08:13.948843 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.948820 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 22:08:13.950747 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.950731 2564 scope.go:117] "RemoveContainer" containerID="abb5d9bd545dc254fe02cf7543098c22d76d36fbd91b22db0198c520a927605b" Sep 10 22:08:13.953881 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.953858 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 22:08:13.958366 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.958350 2564 scope.go:117] "RemoveContainer" containerID="d52e493fcec5931f2283342c4d23a9f40f8d801ff4ef8e65a0312d754a87e556" Sep 10 22:08:13.965141 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.965125 2564 scope.go:117] "RemoveContainer" containerID="3564baae1b1752c1c4b0ceafc3afed66a5cfd8e504912dc3198d75c6cc21f958" Sep 10 22:08:13.971643 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.971626 2564 scope.go:117] "RemoveContainer" containerID="2ba4b556adf46ac19129bc9009150aba50091e4938c5887abe52b0f6878b5d3e" Sep 10 22:08:13.971934 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:08:13.971915 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ba4b556adf46ac19129bc9009150aba50091e4938c5887abe52b0f6878b5d3e\": container with ID starting with 2ba4b556adf46ac19129bc9009150aba50091e4938c5887abe52b0f6878b5d3e not found: ID does not exist" containerID="2ba4b556adf46ac19129bc9009150aba50091e4938c5887abe52b0f6878b5d3e" Sep 10 22:08:13.971985 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.971943 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ba4b556adf46ac19129bc9009150aba50091e4938c5887abe52b0f6878b5d3e"} err="failed to get container status \"2ba4b556adf46ac19129bc9009150aba50091e4938c5887abe52b0f6878b5d3e\": rpc error: code = NotFound desc = could not find container \"2ba4b556adf46ac19129bc9009150aba50091e4938c5887abe52b0f6878b5d3e\": container with ID starting with 2ba4b556adf46ac19129bc9009150aba50091e4938c5887abe52b0f6878b5d3e not found: ID does not exist" Sep 10 22:08:13.971985 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.971960 2564 scope.go:117] "RemoveContainer" containerID="a3f9f7d4a650231a8d54fc111e6c0411f060ce7d74987f9c5a569c8316d4125d" Sep 10 22:08:13.972227 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:08:13.972209 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3f9f7d4a650231a8d54fc111e6c0411f060ce7d74987f9c5a569c8316d4125d\": container with ID starting with a3f9f7d4a650231a8d54fc111e6c0411f060ce7d74987f9c5a569c8316d4125d not found: ID does not exist" containerID="a3f9f7d4a650231a8d54fc111e6c0411f060ce7d74987f9c5a569c8316d4125d" Sep 10 22:08:13.972266 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.972233 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3f9f7d4a650231a8d54fc111e6c0411f060ce7d74987f9c5a569c8316d4125d"} err="failed to get container status \"a3f9f7d4a650231a8d54fc111e6c0411f060ce7d74987f9c5a569c8316d4125d\": rpc error: code = NotFound desc = could not find container \"a3f9f7d4a650231a8d54fc111e6c0411f060ce7d74987f9c5a569c8316d4125d\": container with ID starting with a3f9f7d4a650231a8d54fc111e6c0411f060ce7d74987f9c5a569c8316d4125d not found: ID does not exist" Sep 10 22:08:13.972266 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.972249 2564 scope.go:117] "RemoveContainer" containerID="510ec7b38b032fbb1a6f25f589c53d6573ff417359ac28b683414a4556260f24" Sep 10 22:08:13.972512 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:08:13.972493 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"510ec7b38b032fbb1a6f25f589c53d6573ff417359ac28b683414a4556260f24\": container with ID starting with 510ec7b38b032fbb1a6f25f589c53d6573ff417359ac28b683414a4556260f24 not found: ID does not exist" containerID="510ec7b38b032fbb1a6f25f589c53d6573ff417359ac28b683414a4556260f24" Sep 10 22:08:13.972602 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.972516 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"510ec7b38b032fbb1a6f25f589c53d6573ff417359ac28b683414a4556260f24"} err="failed to get container status \"510ec7b38b032fbb1a6f25f589c53d6573ff417359ac28b683414a4556260f24\": rpc error: code = NotFound desc = could not find container \"510ec7b38b032fbb1a6f25f589c53d6573ff417359ac28b683414a4556260f24\": container with ID starting with 510ec7b38b032fbb1a6f25f589c53d6573ff417359ac28b683414a4556260f24 not found: ID does not exist" Sep 10 22:08:13.972602 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.972530 2564 scope.go:117] "RemoveContainer" containerID="5f6b0090548acfa23acd76ebaaa91990eae84e95bb72186d8d090deba55b7b4b" Sep 10 22:08:13.972764 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:08:13.972746 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f6b0090548acfa23acd76ebaaa91990eae84e95bb72186d8d090deba55b7b4b\": container with ID starting with 5f6b0090548acfa23acd76ebaaa91990eae84e95bb72186d8d090deba55b7b4b not found: ID does not exist" containerID="5f6b0090548acfa23acd76ebaaa91990eae84e95bb72186d8d090deba55b7b4b" Sep 10 22:08:13.972800 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.972768 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f6b0090548acfa23acd76ebaaa91990eae84e95bb72186d8d090deba55b7b4b"} err="failed to get container status \"5f6b0090548acfa23acd76ebaaa91990eae84e95bb72186d8d090deba55b7b4b\": rpc error: code = NotFound desc = could not find container \"5f6b0090548acfa23acd76ebaaa91990eae84e95bb72186d8d090deba55b7b4b\": container with ID starting with 5f6b0090548acfa23acd76ebaaa91990eae84e95bb72186d8d090deba55b7b4b not found: ID does not exist" Sep 10 22:08:13.972800 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.972782 2564 scope.go:117] "RemoveContainer" containerID="abb5d9bd545dc254fe02cf7543098c22d76d36fbd91b22db0198c520a927605b" Sep 10 22:08:13.973043 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:08:13.973027 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abb5d9bd545dc254fe02cf7543098c22d76d36fbd91b22db0198c520a927605b\": container with ID starting with abb5d9bd545dc254fe02cf7543098c22d76d36fbd91b22db0198c520a927605b not found: ID does not exist" containerID="abb5d9bd545dc254fe02cf7543098c22d76d36fbd91b22db0198c520a927605b" Sep 10 22:08:13.973099 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.973048 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb5d9bd545dc254fe02cf7543098c22d76d36fbd91b22db0198c520a927605b"} err="failed to get container status \"abb5d9bd545dc254fe02cf7543098c22d76d36fbd91b22db0198c520a927605b\": rpc error: code = NotFound desc = could not find container \"abb5d9bd545dc254fe02cf7543098c22d76d36fbd91b22db0198c520a927605b\": container with ID starting with abb5d9bd545dc254fe02cf7543098c22d76d36fbd91b22db0198c520a927605b not found: ID does not exist" Sep 10 22:08:13.973099 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.973061 2564 scope.go:117] "RemoveContainer" containerID="d52e493fcec5931f2283342c4d23a9f40f8d801ff4ef8e65a0312d754a87e556" Sep 10 22:08:13.973299 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:08:13.973282 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d52e493fcec5931f2283342c4d23a9f40f8d801ff4ef8e65a0312d754a87e556\": container with ID starting with d52e493fcec5931f2283342c4d23a9f40f8d801ff4ef8e65a0312d754a87e556 not found: ID does not exist" containerID="d52e493fcec5931f2283342c4d23a9f40f8d801ff4ef8e65a0312d754a87e556" Sep 10 22:08:13.973362 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.973306 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d52e493fcec5931f2283342c4d23a9f40f8d801ff4ef8e65a0312d754a87e556"} err="failed to get container status \"d52e493fcec5931f2283342c4d23a9f40f8d801ff4ef8e65a0312d754a87e556\": rpc error: code = NotFound desc = could not find container \"d52e493fcec5931f2283342c4d23a9f40f8d801ff4ef8e65a0312d754a87e556\": container with ID starting with d52e493fcec5931f2283342c4d23a9f40f8d801ff4ef8e65a0312d754a87e556 not found: ID does not exist" Sep 10 22:08:13.973362 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.973325 2564 scope.go:117] "RemoveContainer" containerID="3564baae1b1752c1c4b0ceafc3afed66a5cfd8e504912dc3198d75c6cc21f958" Sep 10 22:08:13.973545 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:08:13.973530 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3564baae1b1752c1c4b0ceafc3afed66a5cfd8e504912dc3198d75c6cc21f958\": container with ID starting with 3564baae1b1752c1c4b0ceafc3afed66a5cfd8e504912dc3198d75c6cc21f958 not found: ID does not exist" containerID="3564baae1b1752c1c4b0ceafc3afed66a5cfd8e504912dc3198d75c6cc21f958" Sep 10 22:08:13.973586 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.973547 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3564baae1b1752c1c4b0ceafc3afed66a5cfd8e504912dc3198d75c6cc21f958"} err="failed to get container status \"3564baae1b1752c1c4b0ceafc3afed66a5cfd8e504912dc3198d75c6cc21f958\": rpc error: code = NotFound desc = could not find container \"3564baae1b1752c1c4b0ceafc3afed66a5cfd8e504912dc3198d75c6cc21f958\": container with ID starting with 3564baae1b1752c1c4b0ceafc3afed66a5cfd8e504912dc3198d75c6cc21f958 not found: ID does not exist" Sep 10 22:08:13.991328 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991308 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 22:08:13.991613 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991601 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="init-config-reloader" Sep 10 22:08:13.991663 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991615 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="init-config-reloader" Sep 10 22:08:13.991663 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991626 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="prom-label-proxy" Sep 10 22:08:13.991663 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991635 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="prom-label-proxy" Sep 10 22:08:13.991663 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991647 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="kube-rbac-proxy-metric" Sep 10 22:08:13.991663 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991655 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="kube-rbac-proxy-metric" Sep 10 22:08:13.991828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991670 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd8bd0ce-a4ba-4134-89aa-224eace7e64d" containerName="registry" Sep 10 22:08:13.991828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991678 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd8bd0ce-a4ba-4134-89aa-224eace7e64d" containerName="registry" Sep 10 22:08:13.991828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991688 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="kube-rbac-proxy" Sep 10 22:08:13.991828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991695 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="kube-rbac-proxy" Sep 10 22:08:13.991828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991702 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="kube-rbac-proxy-web" Sep 10 22:08:13.991828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991707 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="kube-rbac-proxy-web" Sep 10 22:08:13.991828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991716 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="alertmanager" Sep 10 22:08:13.991828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991720 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="alertmanager" Sep 10 22:08:13.991828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991726 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8901fb7d-b8b1-4718-b530-bd480a55a012" containerName="console" Sep 10 22:08:13.991828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991731 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8901fb7d-b8b1-4718-b530-bd480a55a012" containerName="console" Sep 10 22:08:13.991828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991736 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="config-reloader" Sep 10 22:08:13.991828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991741 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="config-reloader" Sep 10 22:08:13.991828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991787 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="prom-label-proxy" Sep 10 22:08:13.991828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991795 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="alertmanager" Sep 10 22:08:13.991828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991802 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="kube-rbac-proxy-web" Sep 10 22:08:13.991828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991808 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd8bd0ce-a4ba-4134-89aa-224eace7e64d" containerName="registry" Sep 10 22:08:13.991828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991813 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="config-reloader" Sep 10 22:08:13.991828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991818 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="kube-rbac-proxy" Sep 10 22:08:13.991828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991822 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f91df0a4-5135-4469-a643-aed828774d7d" containerName="kube-rbac-proxy-metric" Sep 10 22:08:13.991828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.991827 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8901fb7d-b8b1-4718-b530-bd480a55a012" containerName="console" Sep 10 22:08:13.996719 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:13.996704 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.003068 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.002855 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 10 22:08:14.003068 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.002877 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 10 22:08:14.003068 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.002885 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 10 22:08:14.003068 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.002942 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 10 22:08:14.003068 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.002951 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 10 22:08:14.003068 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.002877 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 10 22:08:14.003068 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.002865 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 10 22:08:14.003068 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.002946 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 10 22:08:14.004660 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.003708 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-sc78p\"" Sep 10 22:08:14.011562 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.011535 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 10 22:08:14.021853 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.021831 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 22:08:14.072313 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.072282 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f978096-abb4-4903-8824-7ed487f9d0b3-web-config\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.072444 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.072321 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f978096-abb4-4903-8824-7ed487f9d0b3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.072444 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.072341 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f978096-abb4-4903-8824-7ed487f9d0b3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.072444 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.072387 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f978096-abb4-4903-8824-7ed487f9d0b3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.072444 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.072426 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9f978096-abb4-4903-8824-7ed487f9d0b3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.072577 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.072499 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f978096-abb4-4903-8824-7ed487f9d0b3-config-out\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.072614 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.072576 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f978096-abb4-4903-8824-7ed487f9d0b3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.072614 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.072603 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f978096-abb4-4903-8824-7ed487f9d0b3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.072692 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.072632 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9f978096-abb4-4903-8824-7ed487f9d0b3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.072692 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.072669 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9f978096-abb4-4903-8824-7ed487f9d0b3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.072843 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.072700 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccrlf\" (UniqueName: \"kubernetes.io/projected/9f978096-abb4-4903-8824-7ed487f9d0b3-kube-api-access-ccrlf\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.072843 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.072737 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9f978096-abb4-4903-8824-7ed487f9d0b3-config-volume\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.072843 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.072784 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9f978096-abb4-4903-8824-7ed487f9d0b3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.173688 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.173598 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9f978096-abb4-4903-8824-7ed487f9d0b3-config-volume\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.173688 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.173633 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9f978096-abb4-4903-8824-7ed487f9d0b3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.173945 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.173920 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f978096-abb4-4903-8824-7ed487f9d0b3-web-config\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.174019 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.173978 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f978096-abb4-4903-8824-7ed487f9d0b3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.174019 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.174008 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f978096-abb4-4903-8824-7ed487f9d0b3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.174115 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.174043 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f978096-abb4-4903-8824-7ed487f9d0b3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.174173 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.174110 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9f978096-abb4-4903-8824-7ed487f9d0b3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.174232 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.174182 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f978096-abb4-4903-8824-7ed487f9d0b3-config-out\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.174342 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.174318 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f978096-abb4-4903-8824-7ed487f9d0b3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.175601 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.174558 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f978096-abb4-4903-8824-7ed487f9d0b3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.175601 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.174605 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9f978096-abb4-4903-8824-7ed487f9d0b3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.175601 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.174650 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9f978096-abb4-4903-8824-7ed487f9d0b3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.175601 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.174694 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccrlf\" (UniqueName: \"kubernetes.io/projected/9f978096-abb4-4903-8824-7ed487f9d0b3-kube-api-access-ccrlf\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.175601 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.175308 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9f978096-abb4-4903-8824-7ed487f9d0b3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.175601 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.175548 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f978096-abb4-4903-8824-7ed487f9d0b3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.175970 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.175618 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f978096-abb4-4903-8824-7ed487f9d0b3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.176702 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.176670 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f978096-abb4-4903-8824-7ed487f9d0b3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.176796 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.176738 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9f978096-abb4-4903-8824-7ed487f9d0b3-config-volume\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.176862 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.176800 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9f978096-abb4-4903-8824-7ed487f9d0b3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.177183 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.177162 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f978096-abb4-4903-8824-7ed487f9d0b3-web-config\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.177344 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.177328 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f978096-abb4-4903-8824-7ed487f9d0b3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.177486 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.177465 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f978096-abb4-4903-8824-7ed487f9d0b3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.177621 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.177603 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9f978096-abb4-4903-8824-7ed487f9d0b3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.178212 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.178193 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f978096-abb4-4903-8824-7ed487f9d0b3-config-out\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.178307 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.178291 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9f978096-abb4-4903-8824-7ed487f9d0b3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.184387 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.184363 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccrlf\" (UniqueName: \"kubernetes.io/projected/9f978096-abb4-4903-8824-7ed487f9d0b3-kube-api-access-ccrlf\") pod \"alertmanager-main-0\" (UID: \"9f978096-abb4-4903-8824-7ed487f9d0b3\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.305456 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.305426 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 22:08:14.589986 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.589817 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 22:08:14.592862 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:08:14.592833 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f978096_abb4_4903_8824_7ed487f9d0b3.slice/crio-bbbe1da300d5f03f352cdc33fa39d51b6c9f4f4e91e706ccc6c91ad3ef49b853 WatchSource:0}: Error finding container bbbe1da300d5f03f352cdc33fa39d51b6c9f4f4e91e706ccc6c91ad3ef49b853: Status 404 returned error can't find the container with id bbbe1da300d5f03f352cdc33fa39d51b6c9f4f4e91e706ccc6c91ad3ef49b853 Sep 10 22:08:14.927426 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.927383 2564 generic.go:356] "Generic (PLEG): container finished" podID="9f978096-abb4-4903-8824-7ed487f9d0b3" containerID="b5e8ddb045d64f8c00e24741748eac4c39df7fcf219caee60ac06f4b7687a47b" exitCode=0 Sep 10 22:08:14.927632 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.927474 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f978096-abb4-4903-8824-7ed487f9d0b3","Type":"ContainerDied","Data":"b5e8ddb045d64f8c00e24741748eac4c39df7fcf219caee60ac06f4b7687a47b"} Sep 10 22:08:14.927632 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.927556 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f978096-abb4-4903-8824-7ed487f9d0b3","Type":"ContainerStarted","Data":"bbbe1da300d5f03f352cdc33fa39d51b6c9f4f4e91e706ccc6c91ad3ef49b853"} Sep 10 22:08:14.936095 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:14.936071 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f91df0a4-5135-4469-a643-aed828774d7d" path="/var/lib/kubelet/pods/f91df0a4-5135-4469-a643-aed828774d7d/volumes" Sep 10 22:08:15.935722 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:15.935685 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f978096-abb4-4903-8824-7ed487f9d0b3","Type":"ContainerStarted","Data":"67a38384a8cfc1285a22a30c880ef70bf0269cb2dc76c11170457520bc499c0d"} Sep 10 22:08:15.936140 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:15.935754 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f978096-abb4-4903-8824-7ed487f9d0b3","Type":"ContainerStarted","Data":"c6276e945fd8680375f6996a81b0d4fc78421f924a8f22110c044f88b0769688"} Sep 10 22:08:15.936140 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:15.935782 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f978096-abb4-4903-8824-7ed487f9d0b3","Type":"ContainerStarted","Data":"1baadf1785bc3af9f57edebd15c467ba7d41bbe5c7c857caa1f3836163b1d870"} Sep 10 22:08:15.936140 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:15.935807 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f978096-abb4-4903-8824-7ed487f9d0b3","Type":"ContainerStarted","Data":"6928912e4dc4e5d31c6c2e3bde4e3a89454b2f5d6ebe74482f191e311d90e43c"} Sep 10 22:08:15.936140 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:15.935835 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f978096-abb4-4903-8824-7ed487f9d0b3","Type":"ContainerStarted","Data":"5f4227764c890215bc58d78df353f1f93b543cbd686355d1674dba60f6a9c56c"} Sep 10 22:08:15.936140 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:15.935858 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f978096-abb4-4903-8824-7ed487f9d0b3","Type":"ContainerStarted","Data":"ad55b89f01e71e498f4800ea5302be257b891735042f411cb1107ffd33fa3631"} Sep 10 22:08:15.965510 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:15.965455 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.95071771 podStartE2EDuration="2.965435205s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.717495ms" isStatefulPod=false podCreationTimestamp="2026-09-10 22:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:08:15.963754726 +0000 UTC m=+197.533244713" watchObservedRunningTime="2026-09-10 22:08:15.965435205 +0000 UTC m=+197.534925168" Sep 10 22:08:16.317834 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.317751 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-67fd576bb5-btwmm"] Sep 10 22:08:16.321233 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.321210 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.323782 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.323756 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 10 22:08:16.323946 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.323929 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 10 22:08:16.324072 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.324051 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 10 22:08:16.324123 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.324059 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-9pmjh\"" Sep 10 22:08:16.324318 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.324298 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 10 22:08:16.324426 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.324365 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 10 22:08:16.332712 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.332684 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 10 22:08:16.336168 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.336143 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-67fd576bb5-btwmm"] Sep 10 22:08:16.397170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.397135 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b8d78cd-db2b-48ae-a85e-78621053717b-serving-certs-ca-bundle\") pod \"telemeter-client-67fd576bb5-btwmm\" (UID: \"4b8d78cd-db2b-48ae-a85e-78621053717b\") " pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.397170 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.397173 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b8d78cd-db2b-48ae-a85e-78621053717b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67fd576bb5-btwmm\" (UID: \"4b8d78cd-db2b-48ae-a85e-78621053717b\") " pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.397379 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.397202 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4b8d78cd-db2b-48ae-a85e-78621053717b-secret-telemeter-client\") pod \"telemeter-client-67fd576bb5-btwmm\" (UID: \"4b8d78cd-db2b-48ae-a85e-78621053717b\") " pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.397379 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.397227 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr652\" (UniqueName: \"kubernetes.io/projected/4b8d78cd-db2b-48ae-a85e-78621053717b-kube-api-access-gr652\") pod \"telemeter-client-67fd576bb5-btwmm\" (UID: \"4b8d78cd-db2b-48ae-a85e-78621053717b\") " pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.397379 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.397255 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4b8d78cd-db2b-48ae-a85e-78621053717b-federate-client-tls\") pod \"telemeter-client-67fd576bb5-btwmm\" (UID: \"4b8d78cd-db2b-48ae-a85e-78621053717b\") " pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.397379 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.397272 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b8d78cd-db2b-48ae-a85e-78621053717b-metrics-client-ca\") pod \"telemeter-client-67fd576bb5-btwmm\" (UID: \"4b8d78cd-db2b-48ae-a85e-78621053717b\") " pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.397379 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.397291 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4b8d78cd-db2b-48ae-a85e-78621053717b-telemeter-client-tls\") pod \"telemeter-client-67fd576bb5-btwmm\" (UID: \"4b8d78cd-db2b-48ae-a85e-78621053717b\") " pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.397379 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.397350 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b8d78cd-db2b-48ae-a85e-78621053717b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67fd576bb5-btwmm\" (UID: \"4b8d78cd-db2b-48ae-a85e-78621053717b\") " pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.498524 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.498485 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b8d78cd-db2b-48ae-a85e-78621053717b-serving-certs-ca-bundle\") pod \"telemeter-client-67fd576bb5-btwmm\" (UID: \"4b8d78cd-db2b-48ae-a85e-78621053717b\") " pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.498524 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.498525 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b8d78cd-db2b-48ae-a85e-78621053717b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67fd576bb5-btwmm\" (UID: \"4b8d78cd-db2b-48ae-a85e-78621053717b\") " pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.498766 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.498557 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4b8d78cd-db2b-48ae-a85e-78621053717b-secret-telemeter-client\") pod \"telemeter-client-67fd576bb5-btwmm\" (UID: \"4b8d78cd-db2b-48ae-a85e-78621053717b\") " pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.498766 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.498589 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr652\" (UniqueName: \"kubernetes.io/projected/4b8d78cd-db2b-48ae-a85e-78621053717b-kube-api-access-gr652\") pod \"telemeter-client-67fd576bb5-btwmm\" (UID: \"4b8d78cd-db2b-48ae-a85e-78621053717b\") " pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.498766 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.498629 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4b8d78cd-db2b-48ae-a85e-78621053717b-federate-client-tls\") pod \"telemeter-client-67fd576bb5-btwmm\" (UID: \"4b8d78cd-db2b-48ae-a85e-78621053717b\") " pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.498766 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.498680 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b8d78cd-db2b-48ae-a85e-78621053717b-metrics-client-ca\") pod \"telemeter-client-67fd576bb5-btwmm\" (UID: \"4b8d78cd-db2b-48ae-a85e-78621053717b\") " pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.498766 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.498715 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4b8d78cd-db2b-48ae-a85e-78621053717b-telemeter-client-tls\") pod \"telemeter-client-67fd576bb5-btwmm\" (UID: \"4b8d78cd-db2b-48ae-a85e-78621053717b\") " pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.498766 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.498740 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b8d78cd-db2b-48ae-a85e-78621053717b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67fd576bb5-btwmm\" (UID: \"4b8d78cd-db2b-48ae-a85e-78621053717b\") " pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.499343 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.499309 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b8d78cd-db2b-48ae-a85e-78621053717b-serving-certs-ca-bundle\") pod \"telemeter-client-67fd576bb5-btwmm\" (UID: \"4b8d78cd-db2b-48ae-a85e-78621053717b\") " pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.499466 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.499372 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b8d78cd-db2b-48ae-a85e-78621053717b-metrics-client-ca\") pod \"telemeter-client-67fd576bb5-btwmm\" (UID: \"4b8d78cd-db2b-48ae-a85e-78621053717b\") " pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.499646 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.499622 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b8d78cd-db2b-48ae-a85e-78621053717b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67fd576bb5-btwmm\" (UID: \"4b8d78cd-db2b-48ae-a85e-78621053717b\") " pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.501350 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.501321 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4b8d78cd-db2b-48ae-a85e-78621053717b-secret-telemeter-client\") pod \"telemeter-client-67fd576bb5-btwmm\" (UID: \"4b8d78cd-db2b-48ae-a85e-78621053717b\") " pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.501505 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.501488 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b8d78cd-db2b-48ae-a85e-78621053717b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67fd576bb5-btwmm\" (UID: \"4b8d78cd-db2b-48ae-a85e-78621053717b\") " pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.501597 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.501580 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4b8d78cd-db2b-48ae-a85e-78621053717b-telemeter-client-tls\") pod \"telemeter-client-67fd576bb5-btwmm\" (UID: \"4b8d78cd-db2b-48ae-a85e-78621053717b\") " pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.501747 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.501723 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4b8d78cd-db2b-48ae-a85e-78621053717b-federate-client-tls\") pod \"telemeter-client-67fd576bb5-btwmm\" (UID: \"4b8d78cd-db2b-48ae-a85e-78621053717b\") " pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.509409 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.509387 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr652\" (UniqueName: \"kubernetes.io/projected/4b8d78cd-db2b-48ae-a85e-78621053717b-kube-api-access-gr652\") pod \"telemeter-client-67fd576bb5-btwmm\" (UID: \"4b8d78cd-db2b-48ae-a85e-78621053717b\") " pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.628135 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.628103 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 22:08:16.628466 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.628436 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="kube-rbac-proxy" containerID="cri-o://2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68" gracePeriod=600 Sep 10 22:08:16.628466 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.628455 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="thanos-sidecar" containerID="cri-o://ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7" gracePeriod=600 Sep 10 22:08:16.628466 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.628479 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="config-reloader" containerID="cri-o://8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00" gracePeriod=600 Sep 10 22:08:16.628690 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.628498 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="kube-rbac-proxy-web" containerID="cri-o://489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63" gracePeriod=600 Sep 10 22:08:16.628690 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.628461 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="prometheus" containerID="cri-o://c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5" gracePeriod=600 Sep 10 22:08:16.628690 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.628439 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="kube-rbac-proxy-thanos" containerID="cri-o://be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00" gracePeriod=600 Sep 10 22:08:16.635934 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.634271 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" Sep 10 22:08:16.861721 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.861697 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:16.903695 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.903626 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-web-config\") pod \"1f318b0f-0166-465c-924a-8f246120877e\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " Sep 10 22:08:16.903695 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.903659 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-prometheus-k8s-rulefiles-0\") pod \"1f318b0f-0166-465c-924a-8f246120877e\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " Sep 10 22:08:16.903695 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.903676 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-prometheus-k8s-rulefiles-1\") pod \"1f318b0f-0166-465c-924a-8f246120877e\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " Sep 10 22:08:16.903695 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.903694 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f318b0f-0166-465c-924a-8f246120877e-tls-assets\") pod \"1f318b0f-0166-465c-924a-8f246120877e\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " Sep 10 22:08:16.904048 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.903719 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-thanos-prometheus-http-client-file\") pod \"1f318b0f-0166-465c-924a-8f246120877e\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " Sep 10 22:08:16.904048 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.903886 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-kube-rbac-proxy\") pod \"1f318b0f-0166-465c-924a-8f246120877e\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " Sep 10 22:08:16.904048 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.903935 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-prometheus-trusted-ca-bundle\") pod \"1f318b0f-0166-465c-924a-8f246120877e\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " Sep 10 22:08:16.904048 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.903963 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-prometheus-k8s-tls\") pod \"1f318b0f-0166-465c-924a-8f246120877e\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " Sep 10 22:08:16.904048 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.904008 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"1f318b0f-0166-465c-924a-8f246120877e\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " Sep 10 22:08:16.904048 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.904041 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1f318b0f-0166-465c-924a-8f246120877e-prometheus-k8s-db\") pod \"1f318b0f-0166-465c-924a-8f246120877e\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " Sep 10 22:08:16.904331 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.904073 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-configmap-kubelet-serving-ca-bundle\") pod \"1f318b0f-0166-465c-924a-8f246120877e\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " Sep 10 22:08:16.904331 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.904115 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-config\") pod \"1f318b0f-0166-465c-924a-8f246120877e\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " Sep 10 22:08:16.904331 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.904143 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-configmap-metrics-client-ca\") pod \"1f318b0f-0166-465c-924a-8f246120877e\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " Sep 10 22:08:16.904331 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.904168 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-grpc-tls\") pod \"1f318b0f-0166-465c-924a-8f246120877e\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " Sep 10 22:08:16.904331 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.904210 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"1f318b0f-0166-465c-924a-8f246120877e\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " Sep 10 22:08:16.904331 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.904275 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f318b0f-0166-465c-924a-8f246120877e-config-out\") pod \"1f318b0f-0166-465c-924a-8f246120877e\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " Sep 10 22:08:16.904331 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.904307 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-prometheus-k8s-rulefiles-2\") pod \"1f318b0f-0166-465c-924a-8f246120877e\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " Sep 10 22:08:16.904331 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.904301 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "1f318b0f-0166-465c-924a-8f246120877e" (UID: "1f318b0f-0166-465c-924a-8f246120877e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:08:16.907589 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.905092 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "1f318b0f-0166-465c-924a-8f246120877e" (UID: "1f318b0f-0166-465c-924a-8f246120877e"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:08:16.907589 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.905173 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "1f318b0f-0166-465c-924a-8f246120877e" (UID: "1f318b0f-0166-465c-924a-8f246120877e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:08:16.907589 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.905855 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "1f318b0f-0166-465c-924a-8f246120877e" (UID: "1f318b0f-0166-465c-924a-8f246120877e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:08:16.907589 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.906331 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "1f318b0f-0166-465c-924a-8f246120877e" (UID: "1f318b0f-0166-465c-924a-8f246120877e"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:08:16.907884 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.907350 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "1f318b0f-0166-465c-924a-8f246120877e" (UID: "1f318b0f-0166-465c-924a-8f246120877e"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:08:16.907884 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.904337 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jslch\" (UniqueName: \"kubernetes.io/projected/1f318b0f-0166-465c-924a-8f246120877e-kube-api-access-jslch\") pod \"1f318b0f-0166-465c-924a-8f246120877e\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " Sep 10 22:08:16.907884 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.907678 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-configmap-serving-certs-ca-bundle\") pod \"1f318b0f-0166-465c-924a-8f246120877e\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " Sep 10 22:08:16.907884 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.907714 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-metrics-client-certs\") pod \"1f318b0f-0166-465c-924a-8f246120877e\" (UID: \"1f318b0f-0166-465c-924a-8f246120877e\") " Sep 10 22:08:16.908122 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.908071 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:16.908122 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.908091 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:16.908122 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.908105 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:16.908122 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.908118 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:16.908321 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.908133 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:16.908321 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.908149 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-configmap-metrics-client-ca\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:16.908424 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.908368 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f318b0f-0166-465c-924a-8f246120877e-kube-api-access-jslch" (OuterVolumeSpecName: "kube-api-access-jslch") pod "1f318b0f-0166-465c-924a-8f246120877e" (UID: "1f318b0f-0166-465c-924a-8f246120877e"). InnerVolumeSpecName "kube-api-access-jslch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:08:16.911980 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.908834 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "1f318b0f-0166-465c-924a-8f246120877e" (UID: "1f318b0f-0166-465c-924a-8f246120877e"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:08:16.911980 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.911108 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f318b0f-0166-465c-924a-8f246120877e-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "1f318b0f-0166-465c-924a-8f246120877e" (UID: "1f318b0f-0166-465c-924a-8f246120877e"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 22:08:16.911980 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.911227 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "1f318b0f-0166-465c-924a-8f246120877e" (UID: "1f318b0f-0166-465c-924a-8f246120877e"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:08:16.911980 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.911232 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f318b0f-0166-465c-924a-8f246120877e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1f318b0f-0166-465c-924a-8f246120877e" (UID: "1f318b0f-0166-465c-924a-8f246120877e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:08:16.911980 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.911297 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "1f318b0f-0166-465c-924a-8f246120877e" (UID: "1f318b0f-0166-465c-924a-8f246120877e"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:08:16.911980 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.911787 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "1f318b0f-0166-465c-924a-8f246120877e" (UID: "1f318b0f-0166-465c-924a-8f246120877e"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:08:16.911980 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.911857 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "1f318b0f-0166-465c-924a-8f246120877e" (UID: "1f318b0f-0166-465c-924a-8f246120877e"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:08:16.912484 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.912457 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "1f318b0f-0166-465c-924a-8f246120877e" (UID: "1f318b0f-0166-465c-924a-8f246120877e"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:08:16.912631 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.912600 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "1f318b0f-0166-465c-924a-8f246120877e" (UID: "1f318b0f-0166-465c-924a-8f246120877e"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:08:16.912746 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.912632 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "1f318b0f-0166-465c-924a-8f246120877e" (UID: "1f318b0f-0166-465c-924a-8f246120877e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:08:16.913268 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.913191 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f318b0f-0166-465c-924a-8f246120877e-config-out" (OuterVolumeSpecName: "config-out") pod "1f318b0f-0166-465c-924a-8f246120877e" (UID: "1f318b0f-0166-465c-924a-8f246120877e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 22:08:16.913268 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.913236 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-config" (OuterVolumeSpecName: "config") pod "1f318b0f-0166-465c-924a-8f246120877e" (UID: "1f318b0f-0166-465c-924a-8f246120877e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:08:16.921536 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.921505 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-67fd576bb5-btwmm"] Sep 10 22:08:16.922447 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.922419 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-web-config" (OuterVolumeSpecName: "web-config") pod "1f318b0f-0166-465c-924a-8f246120877e" (UID: "1f318b0f-0166-465c-924a-8f246120877e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:08:16.922995 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:08:16.922973 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8d78cd_db2b_48ae_a85e_78621053717b.slice/crio-b768c83000c7d9e7f35d28ad63a5868c69a58187c309f3d4dae4a88657bb968d WatchSource:0}: Error finding container b768c83000c7d9e7f35d28ad63a5868c69a58187c309f3d4dae4a88657bb968d: Status 404 returned error can't find the container with id b768c83000c7d9e7f35d28ad63a5868c69a58187c309f3d4dae4a88657bb968d Sep 10 22:08:16.944438 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.944402 2564 generic.go:356] "Generic (PLEG): container finished" podID="1f318b0f-0166-465c-924a-8f246120877e" containerID="be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00" exitCode=0 Sep 10 22:08:16.944438 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.944436 2564 generic.go:356] "Generic (PLEG): container finished" podID="1f318b0f-0166-465c-924a-8f246120877e" containerID="2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68" exitCode=0 Sep 10 22:08:16.944860 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.944449 2564 generic.go:356] "Generic (PLEG): container finished" podID="1f318b0f-0166-465c-924a-8f246120877e" containerID="489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63" exitCode=0 Sep 10 22:08:16.944860 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.944443 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f318b0f-0166-465c-924a-8f246120877e","Type":"ContainerDied","Data":"be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00"} Sep 10 22:08:16.944860 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.944462 2564 generic.go:356] "Generic (PLEG): container finished" podID="1f318b0f-0166-465c-924a-8f246120877e" containerID="ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7" exitCode=0 Sep 10 22:08:16.944860 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.944471 2564 generic.go:356] "Generic (PLEG): container finished" podID="1f318b0f-0166-465c-924a-8f246120877e" containerID="8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00" exitCode=0 Sep 10 22:08:16.944860 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.944482 2564 generic.go:356] "Generic (PLEG): container finished" podID="1f318b0f-0166-465c-924a-8f246120877e" containerID="c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5" exitCode=0 Sep 10 22:08:16.944860 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.944489 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:16.944860 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.944529 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f318b0f-0166-465c-924a-8f246120877e","Type":"ContainerDied","Data":"2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68"} Sep 10 22:08:16.944860 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.944571 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f318b0f-0166-465c-924a-8f246120877e","Type":"ContainerDied","Data":"489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63"} Sep 10 22:08:16.944860 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.944609 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f318b0f-0166-465c-924a-8f246120877e","Type":"ContainerDied","Data":"ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7"} Sep 10 22:08:16.944860 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.944636 2564 scope.go:117] "RemoveContainer" containerID="be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00" Sep 10 22:08:16.944860 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.944651 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f318b0f-0166-465c-924a-8f246120877e","Type":"ContainerDied","Data":"8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00"} Sep 10 22:08:16.944860 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.944693 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f318b0f-0166-465c-924a-8f246120877e","Type":"ContainerDied","Data":"c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5"} Sep 10 22:08:16.944860 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.944734 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f318b0f-0166-465c-924a-8f246120877e","Type":"ContainerDied","Data":"662d0bbdd17b1de097ad8ee089d674e1fa925fbc232b71f3ffd8449260d65624"} Sep 10 22:08:16.946325 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.946214 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" event={"ID":"4b8d78cd-db2b-48ae-a85e-78621053717b","Type":"ContainerStarted","Data":"b768c83000c7d9e7f35d28ad63a5868c69a58187c309f3d4dae4a88657bb968d"} Sep 10 22:08:16.952618 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.952602 2564 scope.go:117] "RemoveContainer" containerID="2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68" Sep 10 22:08:16.960910 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.960877 2564 scope.go:117] "RemoveContainer" containerID="489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63" Sep 10 22:08:16.966954 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.966927 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 22:08:16.969395 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.969373 2564 scope.go:117] "RemoveContainer" containerID="ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7" Sep 10 22:08:16.973474 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.973444 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 22:08:16.976570 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.976553 2564 scope.go:117] "RemoveContainer" containerID="8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00" Sep 10 22:08:16.983760 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.983733 2564 scope.go:117] "RemoveContainer" containerID="c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5" Sep 10 22:08:16.990743 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.990726 2564 scope.go:117] "RemoveContainer" containerID="5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd" Sep 10 22:08:16.997766 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.997737 2564 scope.go:117] "RemoveContainer" containerID="be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00" Sep 10 22:08:16.998053 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:08:16.998035 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00\": container with ID starting with be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00 not found: ID does not exist" containerID="be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00" Sep 10 22:08:16.998122 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.998068 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00"} err="failed to get container status \"be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00\": rpc error: code = NotFound desc = could not find container \"be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00\": container with ID starting with be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00 not found: ID does not exist" Sep 10 22:08:16.998122 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.998098 2564 scope.go:117] "RemoveContainer" containerID="2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68" Sep 10 22:08:16.998371 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:08:16.998355 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68\": container with ID starting with 2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68 not found: ID does not exist" containerID="2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68" Sep 10 22:08:16.998410 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.998379 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68"} err="failed to get container status \"2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68\": rpc error: code = NotFound desc = could not find container \"2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68\": container with ID starting with 2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68 not found: ID does not exist" Sep 10 22:08:16.998410 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.998396 2564 scope.go:117] "RemoveContainer" containerID="489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63" Sep 10 22:08:16.998648 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:08:16.998632 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63\": container with ID starting with 489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63 not found: ID does not exist" containerID="489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63" Sep 10 22:08:16.998691 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.998653 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63"} err="failed to get container status \"489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63\": rpc error: code = NotFound desc = could not find container \"489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63\": container with ID starting with 489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63 not found: ID does not exist" Sep 10 22:08:16.998691 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.998667 2564 scope.go:117] "RemoveContainer" containerID="ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7" Sep 10 22:08:16.998892 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:08:16.998876 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7\": container with ID starting with ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7 not found: ID does not exist" containerID="ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7" Sep 10 22:08:16.998987 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.998915 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7"} err="failed to get container status \"ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7\": rpc error: code = NotFound desc = could not find container \"ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7\": container with ID starting with ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7 not found: ID does not exist" Sep 10 22:08:16.998987 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.998934 2564 scope.go:117] "RemoveContainer" containerID="8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00" Sep 10 22:08:16.999171 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:08:16.999157 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00\": container with ID starting with 8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00 not found: ID does not exist" containerID="8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00" Sep 10 22:08:16.999226 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.999176 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00"} err="failed to get container status \"8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00\": rpc error: code = NotFound desc = could not find container \"8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00\": container with ID starting with 8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00 not found: ID does not exist" Sep 10 22:08:16.999226 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.999197 2564 scope.go:117] "RemoveContainer" containerID="c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5" Sep 10 22:08:16.999443 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:08:16.999424 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5\": container with ID starting with c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5 not found: ID does not exist" containerID="c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5" Sep 10 22:08:16.999486 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.999448 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5"} err="failed to get container status \"c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5\": rpc error: code = NotFound desc = could not find container \"c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5\": container with ID starting with c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5 not found: ID does not exist" Sep 10 22:08:16.999486 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.999462 2564 scope.go:117] "RemoveContainer" containerID="5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd" Sep 10 22:08:16.999683 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:08:16.999667 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd\": container with ID starting with 5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd not found: ID does not exist" containerID="5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd" Sep 10 22:08:16.999726 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.999689 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd"} err="failed to get container status \"5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd\": rpc error: code = NotFound desc = could not find container \"5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd\": container with ID starting with 5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd not found: ID does not exist" Sep 10 22:08:16.999726 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.999701 2564 scope.go:117] "RemoveContainer" containerID="be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00" Sep 10 22:08:16.999880 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.999865 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00"} err="failed to get container status \"be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00\": rpc error: code = NotFound desc = could not find container \"be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00\": container with ID starting with be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00 not found: ID does not exist" Sep 10 22:08:16.999880 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:16.999880 2564 scope.go:117] "RemoveContainer" containerID="2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68" Sep 10 22:08:17.000102 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.000086 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68"} err="failed to get container status \"2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68\": rpc error: code = NotFound desc = could not find container \"2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68\": container with ID starting with 2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68 not found: ID does not exist" Sep 10 22:08:17.000158 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.000102 2564 scope.go:117] "RemoveContainer" containerID="489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63" Sep 10 22:08:17.000343 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.000328 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63"} err="failed to get container status \"489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63\": rpc error: code = NotFound desc = could not find container \"489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63\": container with ID starting with 489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63 not found: ID does not exist" Sep 10 22:08:17.000343 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.000342 2564 scope.go:117] "RemoveContainer" containerID="ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7" Sep 10 22:08:17.000540 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.000523 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7"} err="failed to get container status \"ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7\": rpc error: code = NotFound desc = could not find container \"ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7\": container with ID starting with ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7 not found: ID does not exist" Sep 10 22:08:17.000577 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.000540 2564 scope.go:117] "RemoveContainer" containerID="8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00" Sep 10 22:08:17.000736 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.000719 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00"} err="failed to get container status \"8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00\": rpc error: code = NotFound desc = could not find container \"8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00\": container with ID starting with 8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00 not found: ID does not exist" Sep 10 22:08:17.000779 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.000736 2564 scope.go:117] "RemoveContainer" containerID="c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5" Sep 10 22:08:17.000945 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.000928 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5"} err="failed to get container status \"c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5\": rpc error: code = NotFound desc = could not find container \"c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5\": container with ID starting with c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5 not found: ID does not exist" Sep 10 22:08:17.000945 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.000943 2564 scope.go:117] "RemoveContainer" containerID="5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd" Sep 10 22:08:17.001130 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.001114 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd"} err="failed to get container status \"5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd\": rpc error: code = NotFound desc = could not find container \"5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd\": container with ID starting with 5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd not found: ID does not exist" Sep 10 22:08:17.001169 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.001130 2564 scope.go:117] "RemoveContainer" containerID="be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00" Sep 10 22:08:17.001326 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.001303 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00"} err="failed to get container status \"be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00\": rpc error: code = NotFound desc = could not find container \"be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00\": container with ID starting with be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00 not found: ID does not exist" Sep 10 22:08:17.001367 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.001328 2564 scope.go:117] "RemoveContainer" containerID="2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68" Sep 10 22:08:17.001535 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.001521 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68"} err="failed to get container status \"2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68\": rpc error: code = NotFound desc = could not find container \"2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68\": container with ID starting with 2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68 not found: ID does not exist" Sep 10 22:08:17.001573 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.001536 2564 scope.go:117] "RemoveContainer" containerID="489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63" Sep 10 22:08:17.001744 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.001727 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63"} err="failed to get container status \"489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63\": rpc error: code = NotFound desc = could not find container \"489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63\": container with ID starting with 489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63 not found: ID does not exist" Sep 10 22:08:17.001781 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.001744 2564 scope.go:117] "RemoveContainer" containerID="ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7" Sep 10 22:08:17.001956 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.001940 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7"} err="failed to get container status \"ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7\": rpc error: code = NotFound desc = could not find container \"ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7\": container with ID starting with ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7 not found: ID does not exist" Sep 10 22:08:17.001997 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.001959 2564 scope.go:117] "RemoveContainer" containerID="8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00" Sep 10 22:08:17.002166 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.002151 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00"} err="failed to get container status \"8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00\": rpc error: code = NotFound desc = could not find container \"8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00\": container with ID starting with 8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00 not found: ID does not exist" Sep 10 22:08:17.002208 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.002166 2564 scope.go:117] "RemoveContainer" containerID="c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5" Sep 10 22:08:17.002361 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.002342 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5"} err="failed to get container status \"c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5\": rpc error: code = NotFound desc = could not find container \"c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5\": container with ID starting with c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5 not found: ID does not exist" Sep 10 22:08:17.002405 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.002362 2564 scope.go:117] "RemoveContainer" containerID="5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd" Sep 10 22:08:17.002616 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.002593 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd"} err="failed to get container status \"5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd\": rpc error: code = NotFound desc = could not find container \"5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd\": container with ID starting with 5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd not found: ID does not exist" Sep 10 22:08:17.002616 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.002614 2564 scope.go:117] "RemoveContainer" containerID="be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00" Sep 10 22:08:17.003042 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.002984 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00"} err="failed to get container status \"be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00\": rpc error: code = NotFound desc = could not find container \"be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00\": container with ID starting with be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00 not found: ID does not exist" Sep 10 22:08:17.003042 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.003011 2564 scope.go:117] "RemoveContainer" containerID="2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68" Sep 10 22:08:17.003281 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.003262 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68"} err="failed to get container status \"2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68\": rpc error: code = NotFound desc = could not find container \"2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68\": container with ID starting with 2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68 not found: ID does not exist" Sep 10 22:08:17.003346 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.003291 2564 scope.go:117] "RemoveContainer" containerID="489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63" Sep 10 22:08:17.003559 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.003540 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63"} err="failed to get container status \"489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63\": rpc error: code = NotFound desc = could not find container \"489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63\": container with ID starting with 489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63 not found: ID does not exist" Sep 10 22:08:17.003742 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.003559 2564 scope.go:117] "RemoveContainer" containerID="ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7" Sep 10 22:08:17.003830 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.003809 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7"} err="failed to get container status \"ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7\": rpc error: code = NotFound desc = could not find container \"ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7\": container with ID starting with ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7 not found: ID does not exist" Sep 10 22:08:17.003883 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.003833 2564 scope.go:117] "RemoveContainer" containerID="8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00" Sep 10 22:08:17.004096 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.004077 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00"} err="failed to get container status \"8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00\": rpc error: code = NotFound desc = could not find container \"8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00\": container with ID starting with 8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00 not found: ID does not exist" Sep 10 22:08:17.004180 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.004098 2564 scope.go:117] "RemoveContainer" containerID="c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5" Sep 10 22:08:17.004360 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.004344 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5"} err="failed to get container status \"c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5\": rpc error: code = NotFound desc = could not find container \"c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5\": container with ID starting with c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5 not found: ID does not exist" Sep 10 22:08:17.004418 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.004360 2564 scope.go:117] "RemoveContainer" containerID="5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd" Sep 10 22:08:17.004578 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.004563 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd"} err="failed to get container status \"5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd\": rpc error: code = NotFound desc = could not find container \"5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd\": container with ID starting with 5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd not found: ID does not exist" Sep 10 22:08:17.004642 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.004580 2564 scope.go:117] "RemoveContainer" containerID="be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00" Sep 10 22:08:17.004831 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.004813 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00"} err="failed to get container status \"be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00\": rpc error: code = NotFound desc = could not find container \"be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00\": container with ID starting with be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00 not found: ID does not exist" Sep 10 22:08:17.004874 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.004832 2564 scope.go:117] "RemoveContainer" containerID="2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68" Sep 10 22:08:17.004947 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.004878 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 22:08:17.005125 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005106 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68"} err="failed to get container status \"2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68\": rpc error: code = NotFound desc = could not find container \"2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68\": container with ID starting with 2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68 not found: ID does not exist" Sep 10 22:08:17.005167 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005126 2564 scope.go:117] "RemoveContainer" containerID="489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63" Sep 10 22:08:17.005244 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005230 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="thanos-sidecar" Sep 10 22:08:17.005307 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005246 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="thanos-sidecar" Sep 10 22:08:17.005307 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005258 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="kube-rbac-proxy-thanos" Sep 10 22:08:17.005307 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005266 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="kube-rbac-proxy-thanos" Sep 10 22:08:17.005307 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005278 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="init-config-reloader" Sep 10 22:08:17.005307 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005285 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="init-config-reloader" Sep 10 22:08:17.005307 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005299 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="kube-rbac-proxy-web" Sep 10 22:08:17.005307 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005304 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="kube-rbac-proxy-web" Sep 10 22:08:17.005307 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005309 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="kube-rbac-proxy" Sep 10 22:08:17.005579 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005314 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="kube-rbac-proxy" Sep 10 22:08:17.005579 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005320 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="prometheus" Sep 10 22:08:17.005579 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005325 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="prometheus" Sep 10 22:08:17.005579 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005324 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63"} err="failed to get container status \"489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63\": rpc error: code = NotFound desc = could not find container \"489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63\": container with ID starting with 489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63 not found: ID does not exist" Sep 10 22:08:17.005579 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005336 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="config-reloader" Sep 10 22:08:17.005579 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005338 2564 scope.go:117] "RemoveContainer" containerID="ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7" Sep 10 22:08:17.005579 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005341 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="config-reloader" Sep 10 22:08:17.005579 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005451 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="kube-rbac-proxy-web" Sep 10 22:08:17.005579 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005460 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="kube-rbac-proxy" Sep 10 22:08:17.005579 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005466 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="kube-rbac-proxy-thanos" Sep 10 22:08:17.005579 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005474 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="config-reloader" Sep 10 22:08:17.005579 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005481 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="prometheus" Sep 10 22:08:17.005579 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005486 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f318b0f-0166-465c-924a-8f246120877e" containerName="thanos-sidecar" Sep 10 22:08:17.006022 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005606 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7"} err="failed to get container status \"ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7\": rpc error: code = NotFound desc = could not find container \"ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7\": container with ID starting with ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7 not found: ID does not exist" Sep 10 22:08:17.006022 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005621 2564 scope.go:117] "RemoveContainer" containerID="8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00" Sep 10 22:08:17.006022 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005836 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00"} err="failed to get container status \"8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00\": rpc error: code = NotFound desc = could not find container \"8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00\": container with ID starting with 8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00 not found: ID does not exist" Sep 10 22:08:17.006022 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.005854 2564 scope.go:117] "RemoveContainer" containerID="c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5" Sep 10 22:08:17.006201 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.006080 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5"} err="failed to get container status \"c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5\": rpc error: code = NotFound desc = could not find container \"c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5\": container with ID starting with c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5 not found: ID does not exist" Sep 10 22:08:17.006201 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.006096 2564 scope.go:117] "RemoveContainer" containerID="5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd" Sep 10 22:08:17.006349 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.006331 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd"} err="failed to get container status \"5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd\": rpc error: code = NotFound desc = could not find container \"5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd\": container with ID starting with 5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd not found: ID does not exist" Sep 10 22:08:17.006389 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.006350 2564 scope.go:117] "RemoveContainer" containerID="be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00" Sep 10 22:08:17.006570 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.006556 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00"} err="failed to get container status \"be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00\": rpc error: code = NotFound desc = could not find container \"be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00\": container with ID starting with be873aaef71511ef8394942f0847062c0dcb6fe3b0477e67e4637898e1485f00 not found: ID does not exist" Sep 10 22:08:17.006629 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.006570 2564 scope.go:117] "RemoveContainer" containerID="2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68" Sep 10 22:08:17.006787 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.006771 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68"} err="failed to get container status \"2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68\": rpc error: code = NotFound desc = could not find container \"2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68\": container with ID starting with 2a072fe2d3ab82a5a19fc6dbef120618c8f71d0a97044c39557bc0e7a5bfee68 not found: ID does not exist" Sep 10 22:08:17.006832 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.006787 2564 scope.go:117] "RemoveContainer" containerID="489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63" Sep 10 22:08:17.007030 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.007011 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63"} err="failed to get container status \"489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63\": rpc error: code = NotFound desc = could not find container \"489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63\": container with ID starting with 489a8355ab701186d1a012f24232f9f13961588eb54e84a62877079331c89b63 not found: ID does not exist" Sep 10 22:08:17.007076 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.007030 2564 scope.go:117] "RemoveContainer" containerID="ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7" Sep 10 22:08:17.007265 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.007249 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7"} err="failed to get container status \"ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7\": rpc error: code = NotFound desc = could not find container \"ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7\": container with ID starting with ff28d81963efb1ed412264e1cccac284d8e7d093fdba7d582f314a6f15f4b8b7 not found: ID does not exist" Sep 10 22:08:17.007308 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.007265 2564 scope.go:117] "RemoveContainer" containerID="8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00" Sep 10 22:08:17.007491 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.007474 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00"} err="failed to get container status \"8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00\": rpc error: code = NotFound desc = could not find container \"8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00\": container with ID starting with 8994530536b5edbe5cf6116a6e27a5ac6170a7c2d023fe8a875b1fe374a92e00 not found: ID does not exist" Sep 10 22:08:17.007527 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.007491 2564 scope.go:117] "RemoveContainer" containerID="c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5" Sep 10 22:08:17.007746 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.007724 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5"} err="failed to get container status \"c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5\": rpc error: code = NotFound desc = could not find container \"c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5\": container with ID starting with c4093607726a4552edb523bed3078cbb99dc10cf2061843c57dbebe8f3fe75f5 not found: ID does not exist" Sep 10 22:08:17.007746 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.007744 2564 scope.go:117] "RemoveContainer" containerID="5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd" Sep 10 22:08:17.007981 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.007963 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd"} err="failed to get container status \"5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd\": rpc error: code = NotFound desc = could not find container \"5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd\": container with ID starting with 5b46dd31aa203c3fa97e7a7caee7f0e80b143a9e962aa0cda4e38f640b7c00cd not found: ID does not exist" Sep 10 22:08:17.009920 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.009883 2564 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-config\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:17.009920 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.009917 2564 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-grpc-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:17.010029 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.009930 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:17.010029 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.009939 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f318b0f-0166-465c-924a-8f246120877e-config-out\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:17.010029 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.009947 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jslch\" (UniqueName: \"kubernetes.io/projected/1f318b0f-0166-465c-924a-8f246120877e-kube-api-access-jslch\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:17.010029 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.009955 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f318b0f-0166-465c-924a-8f246120877e-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:17.010029 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.009964 2564 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-metrics-client-certs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:17.010029 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.009972 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-web-config\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:17.010029 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.009980 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f318b0f-0166-465c-924a-8f246120877e-tls-assets\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:17.010029 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.009988 2564 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:17.010029 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.009996 2564 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-kube-rbac-proxy\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:17.010029 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.010003 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:17.010029 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.010013 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1f318b0f-0166-465c-924a-8f246120877e-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:17.010029 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.010022 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1f318b0f-0166-465c-924a-8f246120877e-prometheus-k8s-db\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:08:17.010699 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.010682 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.013504 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.013487 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 10 22:08:17.014017 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.014000 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fv1dc3gjvdb89\"" Sep 10 22:08:17.014017 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.014009 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 10 22:08:17.014168 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.014076 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 10 22:08:17.014259 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.014244 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9f4rs\"" Sep 10 22:08:17.014560 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.014544 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 10 22:08:17.014684 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.014567 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 10 22:08:17.014684 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.014567 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 10 22:08:17.014855 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.014842 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 10 22:08:17.015347 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.015331 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 10 22:08:17.015429 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.015381 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 10 22:08:17.015429 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.015397 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 10 22:08:17.015605 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.015591 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 10 22:08:17.015974 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.015940 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 10 22:08:17.018818 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.018798 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 10 22:08:17.022722 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.022701 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 10 22:08:17.029935 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.029912 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 22:08:17.110590 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.110556 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.110759 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.110605 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.110759 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.110628 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-web-config\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.110759 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.110675 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.110759 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.110694 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.110759 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.110717 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.110759 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.110754 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.111004 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.110771 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.111004 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.110790 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.111004 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.110831 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.111004 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.110850 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.111004 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.110885 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.111004 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.110930 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.111004 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.110955 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-config-out\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.111004 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.110975 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.111004 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.110989 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.111338 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.111009 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpcbb\" (UniqueName: \"kubernetes.io/projected/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-kube-api-access-hpcbb\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.111338 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.111036 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-config\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.111338 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.111090 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.111338 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.111127 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.211628 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.211527 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.211628 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.211559 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.211628 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.211578 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.211628 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.211610 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.211872 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.211658 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.212350 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.212000 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.212350 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.212076 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.212350 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.212109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-config-out\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.212350 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.212344 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.212617 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.212399 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.214180 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.213123 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.214180 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.213421 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.214180 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.213752 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.214180 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.214025 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.214561 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.214379 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpcbb\" (UniqueName: \"kubernetes.io/projected/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-kube-api-access-hpcbb\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.214561 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.214426 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-config\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.214561 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.214463 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.214561 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.214490 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.214753 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.214536 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.214753 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.214649 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.214753 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.214689 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-web-config\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.215807 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.214730 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.215807 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.214810 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.215807 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.214846 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.216300 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.216043 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.216300 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.216183 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-config-out\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.216688 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.216665 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.218008 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.217982 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.218331 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.218306 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.218839 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.218810 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.219440 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.219419 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.220968 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.220203 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.220968 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.220664 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.223565 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.221688 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.223565 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.221697 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.223565 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.221918 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.223565 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.222124 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-config\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.223565 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.222283 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.223565 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.223253 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-web-config\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.228544 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.228522 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpcbb\" (UniqueName: \"kubernetes.io/projected/0c8e571e-6e80-4e58-ac7b-8e09b9d52218-kube-api-access-hpcbb\") pod \"prometheus-k8s-0\" (UID: \"0c8e571e-6e80-4e58-ac7b-8e09b9d52218\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.320937 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.320877 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:08:17.604800 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.604771 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 22:08:17.606971 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:08:17.606940 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c8e571e_6e80_4e58_ac7b_8e09b9d52218.slice/crio-ae87fcb3c7a90910e4c41ed7069ee8873774816c33c04d70b799ffa06f937797 WatchSource:0}: Error finding container ae87fcb3c7a90910e4c41ed7069ee8873774816c33c04d70b799ffa06f937797: Status 404 returned error can't find the container with id ae87fcb3c7a90910e4c41ed7069ee8873774816c33c04d70b799ffa06f937797 Sep 10 22:08:17.951935 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.951879 2564 generic.go:356] "Generic (PLEG): container finished" podID="0c8e571e-6e80-4e58-ac7b-8e09b9d52218" containerID="a720a5432d2d00c90f971c1c9766e9b347f6fd620ba5d117b8b2dab4c37cfd0b" exitCode=0 Sep 10 22:08:17.952344 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.951972 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c8e571e-6e80-4e58-ac7b-8e09b9d52218","Type":"ContainerDied","Data":"a720a5432d2d00c90f971c1c9766e9b347f6fd620ba5d117b8b2dab4c37cfd0b"} Sep 10 22:08:17.952344 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:17.952058 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c8e571e-6e80-4e58-ac7b-8e09b9d52218","Type":"ContainerStarted","Data":"ae87fcb3c7a90910e4c41ed7069ee8873774816c33c04d70b799ffa06f937797"} Sep 10 22:08:18.937517 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:18.937482 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f318b0f-0166-465c-924a-8f246120877e" path="/var/lib/kubelet/pods/1f318b0f-0166-465c-924a-8f246120877e/volumes" Sep 10 22:08:18.964521 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:18.964488 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c8e571e-6e80-4e58-ac7b-8e09b9d52218","Type":"ContainerStarted","Data":"d816d1b269d583e8ba39a0272358903a48a8d05fc909bb14cbb171cacf809f84"} Sep 10 22:08:18.964982 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:18.964561 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c8e571e-6e80-4e58-ac7b-8e09b9d52218","Type":"ContainerStarted","Data":"945c853963bfaebfced451f8ffcd9c3de3952840a6e5d27e60eeb605a846d3b5"} Sep 10 22:08:18.964982 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:18.964600 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c8e571e-6e80-4e58-ac7b-8e09b9d52218","Type":"ContainerStarted","Data":"b7703fc58bff0bf3dc7ebe6580176c1d22fb8c3d045ebeb2e9737df0c79e9e12"} Sep 10 22:08:18.964982 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:18.964639 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c8e571e-6e80-4e58-ac7b-8e09b9d52218","Type":"ContainerStarted","Data":"b8a1d19d85d6656948ae4a1fbcfe09184145801b1d49ee2cc76b05559def4a97"} Sep 10 22:08:18.964982 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:18.964681 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c8e571e-6e80-4e58-ac7b-8e09b9d52218","Type":"ContainerStarted","Data":"21c0303f53de0ac3a890d7e6a5ac9b653d70313dce17725cac947ec2d17e4237"} Sep 10 22:08:18.964982 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:18.964716 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c8e571e-6e80-4e58-ac7b-8e09b9d52218","Type":"ContainerStarted","Data":"e62dbb57dbd335d58281a787c4d853ca1e06d65954eb21067d7f01212347ad34"} Sep 10 22:08:19.002612 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:19.002560 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.980051082 podStartE2EDuration="3.002543246s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.492164ms" isStatefulPod=false podCreationTimestamp="2026-09-10 22:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:08:18.999417603 +0000 UTC m=+200.568907606" watchObservedRunningTime="2026-09-10 22:08:19.002543246 +0000 UTC m=+200.572033210" Sep 10 22:08:19.970371 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:19.970334 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" event={"ID":"4b8d78cd-db2b-48ae-a85e-78621053717b","Type":"ContainerStarted","Data":"2c17b2885801ebd0a034145b26b17995883e509b412c6f0e4a7e1aaf6fda81fb"} Sep 10 22:08:19.970756 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:19.970387 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" event={"ID":"4b8d78cd-db2b-48ae-a85e-78621053717b","Type":"ContainerStarted","Data":"d3f7fb6acec7caaba09f57b57d77c167383fe577ae8bdc4af97d60e9f1be3373"} Sep 10 22:08:19.970756 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:19.970406 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" event={"ID":"4b8d78cd-db2b-48ae-a85e-78621053717b","Type":"ContainerStarted","Data":"0ca1bc9c910f7d94f287b4abbc5fafde0e728df0442793ab2735e3e8dc9871bf"} Sep 10 22:08:19.996788 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:19.996744 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-67fd576bb5-btwmm" podStartSLOduration=1.762467563 podStartE2EDuration="3.996729324s" totalImagesPullingTime="2.234261761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:08:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:08:19.994754789 +0000 UTC m=+201.564244778" watchObservedRunningTime="2026-09-10 22:08:19.996729324 +0000 UTC m=+201.566219287" Sep 10 22:08:22.321034 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:08:22.320993 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:09:17.322124 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:09:17.322086 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:09:17.338116 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:09:17.338089 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:09:18.223592 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:09:18.223551 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 22:09:58.866100 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:09:58.866076 2564 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 22:11:36.005514 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.005480 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq"] Sep 10 22:11:36.008838 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.008822 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq" Sep 10 22:11:36.011863 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.011840 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 22:11:36.011863 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.011849 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 22:11:36.012334 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.012308 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 22:11:36.012609 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.012594 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 22:11:36.016326 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.016306 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4v29l\"" Sep 10 22:11:36.037017 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.036994 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq"] Sep 10 22:11:36.049530 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.049511 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cee352bc-150c-4c82-901a-e5871cbf837d-tmp\") pod \"opendatahub-operator-controller-manager-995c55587-hppcq\" (UID: \"cee352bc-150c-4c82-901a-e5871cbf837d\") " pod="opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq" Sep 10 22:11:36.049613 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.049548 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cee352bc-150c-4c82-901a-e5871cbf837d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-995c55587-hppcq\" (UID: \"cee352bc-150c-4c82-901a-e5871cbf837d\") " pod="opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq" Sep 10 22:11:36.049613 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.049589 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cee352bc-150c-4c82-901a-e5871cbf837d-manifests\") pod \"opendatahub-operator-controller-manager-995c55587-hppcq\" (UID: \"cee352bc-150c-4c82-901a-e5871cbf837d\") " pod="opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq" Sep 10 22:11:36.049683 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.049613 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cee352bc-150c-4c82-901a-e5871cbf837d-webhook-cert\") pod \"opendatahub-operator-controller-manager-995c55587-hppcq\" (UID: \"cee352bc-150c-4c82-901a-e5871cbf837d\") " pod="opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq" Sep 10 22:11:36.049683 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.049654 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9j7f\" (UniqueName: \"kubernetes.io/projected/cee352bc-150c-4c82-901a-e5871cbf837d-kube-api-access-q9j7f\") pod \"opendatahub-operator-controller-manager-995c55587-hppcq\" (UID: \"cee352bc-150c-4c82-901a-e5871cbf837d\") " pod="opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq" Sep 10 22:11:36.150487 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.150443 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cee352bc-150c-4c82-901a-e5871cbf837d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-995c55587-hppcq\" (UID: \"cee352bc-150c-4c82-901a-e5871cbf837d\") " pod="opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq" Sep 10 22:11:36.150641 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.150506 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cee352bc-150c-4c82-901a-e5871cbf837d-manifests\") pod \"opendatahub-operator-controller-manager-995c55587-hppcq\" (UID: \"cee352bc-150c-4c82-901a-e5871cbf837d\") " pod="opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq" Sep 10 22:11:36.150641 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.150543 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cee352bc-150c-4c82-901a-e5871cbf837d-webhook-cert\") pod \"opendatahub-operator-controller-manager-995c55587-hppcq\" (UID: \"cee352bc-150c-4c82-901a-e5871cbf837d\") " pod="opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq" Sep 10 22:11:36.150641 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.150576 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9j7f\" (UniqueName: \"kubernetes.io/projected/cee352bc-150c-4c82-901a-e5871cbf837d-kube-api-access-q9j7f\") pod \"opendatahub-operator-controller-manager-995c55587-hppcq\" (UID: \"cee352bc-150c-4c82-901a-e5871cbf837d\") " pod="opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq" Sep 10 22:11:36.150641 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.150601 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cee352bc-150c-4c82-901a-e5871cbf837d-tmp\") pod \"opendatahub-operator-controller-manager-995c55587-hppcq\" (UID: \"cee352bc-150c-4c82-901a-e5871cbf837d\") " pod="opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq" Sep 10 22:11:36.151029 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.151003 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cee352bc-150c-4c82-901a-e5871cbf837d-manifests\") pod \"opendatahub-operator-controller-manager-995c55587-hppcq\" (UID: \"cee352bc-150c-4c82-901a-e5871cbf837d\") " pod="opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq" Sep 10 22:11:36.152924 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.152879 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cee352bc-150c-4c82-901a-e5871cbf837d-tmp\") pod \"opendatahub-operator-controller-manager-995c55587-hppcq\" (UID: \"cee352bc-150c-4c82-901a-e5871cbf837d\") " pod="opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq" Sep 10 22:11:36.153028 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.152975 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cee352bc-150c-4c82-901a-e5871cbf837d-webhook-cert\") pod \"opendatahub-operator-controller-manager-995c55587-hppcq\" (UID: \"cee352bc-150c-4c82-901a-e5871cbf837d\") " pod="opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq" Sep 10 22:11:36.153087 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.153027 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cee352bc-150c-4c82-901a-e5871cbf837d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-995c55587-hppcq\" (UID: \"cee352bc-150c-4c82-901a-e5871cbf837d\") " pod="opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq" Sep 10 22:11:36.162494 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.162474 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9j7f\" (UniqueName: \"kubernetes.io/projected/cee352bc-150c-4c82-901a-e5871cbf837d-kube-api-access-q9j7f\") pod \"opendatahub-operator-controller-manager-995c55587-hppcq\" (UID: \"cee352bc-150c-4c82-901a-e5871cbf837d\") " pod="opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq" Sep 10 22:11:36.318671 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.318604 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq" Sep 10 22:11:36.596412 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.596383 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq"] Sep 10 22:11:36.597955 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:11:36.597924 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee352bc_150c_4c82_901a_e5871cbf837d.slice/crio-bf623402d6b07722728f422daedc16e7e2d90a0559c12d34e8e22806a0ef9d0f WatchSource:0}: Error finding container bf623402d6b07722728f422daedc16e7e2d90a0559c12d34e8e22806a0ef9d0f: Status 404 returned error can't find the container with id bf623402d6b07722728f422daedc16e7e2d90a0559c12d34e8e22806a0ef9d0f Sep 10 22:11:36.599671 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.599652 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 22:11:36.769585 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:36.769547 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq" event={"ID":"cee352bc-150c-4c82-901a-e5871cbf837d","Type":"ContainerStarted","Data":"bf623402d6b07722728f422daedc16e7e2d90a0559c12d34e8e22806a0ef9d0f"} Sep 10 22:11:39.786002 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:39.785894 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq" event={"ID":"cee352bc-150c-4c82-901a-e5871cbf837d","Type":"ContainerStarted","Data":"c2b181468860412012dddc9d699fd553b11137e80790193fa2f7bea061d8e128"} Sep 10 22:11:40.792553 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:40.792518 2564 generic.go:356] "Generic (PLEG): container finished" podID="cee352bc-150c-4c82-901a-e5871cbf837d" containerID="c2b181468860412012dddc9d699fd553b11137e80790193fa2f7bea061d8e128" exitCode=0 Sep 10 22:11:40.792928 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:40.792589 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq" event={"ID":"cee352bc-150c-4c82-901a-e5871cbf837d","Type":"ContainerDied","Data":"c2b181468860412012dddc9d699fd553b11137e80790193fa2f7bea061d8e128"} Sep 10 22:11:41.799729 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:41.799687 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq" event={"ID":"cee352bc-150c-4c82-901a-e5871cbf837d","Type":"ContainerStarted","Data":"3f4992da140a8fa9dc1011839e3f235a3bedd93c4fd64d2a45cdc0cdeae0deec"} Sep 10 22:11:41.800161 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:41.799810 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq" Sep 10 22:11:41.826065 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:41.826021 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq" podStartSLOduration=2.611709055 podStartE2EDuration="6.826004872s" totalImagesPullingTime="3.188814549s" totalInitContainerRuntime="1.025481268s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:11:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:11:41.824260058 +0000 UTC m=+403.393750024" watchObservedRunningTime="2026-09-10 22:11:41.826004872 +0000 UTC m=+403.395494836" Sep 10 22:11:52.806850 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:52.806819 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-995c55587-hppcq" Sep 10 22:11:56.159911 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:56.159859 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-578b4977d8-d9snc"] Sep 10 22:11:56.166755 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:56.166733 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-578b4977d8-d9snc" Sep 10 22:11:56.170462 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:56.170436 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 10 22:11:56.170735 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:56.170443 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 10 22:11:56.170834 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:56.170477 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 10 22:11:56.170834 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:56.170500 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-52gnx\"" Sep 10 22:11:56.170978 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:56.170521 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 10 22:11:56.170978 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:56.170590 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 10 22:11:56.172473 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:56.172452 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-578b4977d8-d9snc"] Sep 10 22:11:56.334037 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:56.333978 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fb31c55-8407-49ea-9b6f-044e2cf97e12-metrics-cert\") pod \"lws-controller-manager-578b4977d8-d9snc\" (UID: \"7fb31c55-8407-49ea-9b6f-044e2cf97e12\") " pod="openshift-lws-operator/lws-controller-manager-578b4977d8-d9snc" Sep 10 22:11:56.334259 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:56.334131 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fb31c55-8407-49ea-9b6f-044e2cf97e12-cert\") pod \"lws-controller-manager-578b4977d8-d9snc\" (UID: \"7fb31c55-8407-49ea-9b6f-044e2cf97e12\") " pod="openshift-lws-operator/lws-controller-manager-578b4977d8-d9snc" Sep 10 22:11:56.334259 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:56.334175 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9rz2\" (UniqueName: \"kubernetes.io/projected/7fb31c55-8407-49ea-9b6f-044e2cf97e12-kube-api-access-r9rz2\") pod \"lws-controller-manager-578b4977d8-d9snc\" (UID: \"7fb31c55-8407-49ea-9b6f-044e2cf97e12\") " pod="openshift-lws-operator/lws-controller-manager-578b4977d8-d9snc" Sep 10 22:11:56.334259 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:56.334236 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7fb31c55-8407-49ea-9b6f-044e2cf97e12-manager-config\") pod \"lws-controller-manager-578b4977d8-d9snc\" (UID: \"7fb31c55-8407-49ea-9b6f-044e2cf97e12\") " pod="openshift-lws-operator/lws-controller-manager-578b4977d8-d9snc" Sep 10 22:11:56.435172 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:56.435070 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fb31c55-8407-49ea-9b6f-044e2cf97e12-metrics-cert\") pod \"lws-controller-manager-578b4977d8-d9snc\" (UID: \"7fb31c55-8407-49ea-9b6f-044e2cf97e12\") " pod="openshift-lws-operator/lws-controller-manager-578b4977d8-d9snc" Sep 10 22:11:56.435172 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:56.435158 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fb31c55-8407-49ea-9b6f-044e2cf97e12-cert\") pod \"lws-controller-manager-578b4977d8-d9snc\" (UID: \"7fb31c55-8407-49ea-9b6f-044e2cf97e12\") " pod="openshift-lws-operator/lws-controller-manager-578b4977d8-d9snc" Sep 10 22:11:56.435417 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:56.435189 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9rz2\" (UniqueName: \"kubernetes.io/projected/7fb31c55-8407-49ea-9b6f-044e2cf97e12-kube-api-access-r9rz2\") pod \"lws-controller-manager-578b4977d8-d9snc\" (UID: \"7fb31c55-8407-49ea-9b6f-044e2cf97e12\") " pod="openshift-lws-operator/lws-controller-manager-578b4977d8-d9snc" Sep 10 22:11:56.435417 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:56.435224 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7fb31c55-8407-49ea-9b6f-044e2cf97e12-manager-config\") pod \"lws-controller-manager-578b4977d8-d9snc\" (UID: \"7fb31c55-8407-49ea-9b6f-044e2cf97e12\") " pod="openshift-lws-operator/lws-controller-manager-578b4977d8-d9snc" Sep 10 22:11:56.435978 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:56.435955 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7fb31c55-8407-49ea-9b6f-044e2cf97e12-manager-config\") pod \"lws-controller-manager-578b4977d8-d9snc\" (UID: \"7fb31c55-8407-49ea-9b6f-044e2cf97e12\") " pod="openshift-lws-operator/lws-controller-manager-578b4977d8-d9snc" Sep 10 22:11:56.437634 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:56.437614 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fb31c55-8407-49ea-9b6f-044e2cf97e12-metrics-cert\") pod \"lws-controller-manager-578b4977d8-d9snc\" (UID: \"7fb31c55-8407-49ea-9b6f-044e2cf97e12\") " pod="openshift-lws-operator/lws-controller-manager-578b4977d8-d9snc" Sep 10 22:11:56.437711 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:56.437664 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fb31c55-8407-49ea-9b6f-044e2cf97e12-cert\") pod \"lws-controller-manager-578b4977d8-d9snc\" (UID: \"7fb31c55-8407-49ea-9b6f-044e2cf97e12\") " pod="openshift-lws-operator/lws-controller-manager-578b4977d8-d9snc" Sep 10 22:11:56.443094 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:56.443076 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9rz2\" (UniqueName: \"kubernetes.io/projected/7fb31c55-8407-49ea-9b6f-044e2cf97e12-kube-api-access-r9rz2\") pod \"lws-controller-manager-578b4977d8-d9snc\" (UID: \"7fb31c55-8407-49ea-9b6f-044e2cf97e12\") " pod="openshift-lws-operator/lws-controller-manager-578b4977d8-d9snc" Sep 10 22:11:56.477175 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:56.477147 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-578b4977d8-d9snc" Sep 10 22:11:56.747012 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:56.746987 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-578b4977d8-d9snc"] Sep 10 22:11:56.748862 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:11:56.748835 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb31c55_8407_49ea_9b6f_044e2cf97e12.slice/crio-5ed61a99a29a5fe3572311cbe1bb02ef860e7a46d56fef336887e869c490434d WatchSource:0}: Error finding container 5ed61a99a29a5fe3572311cbe1bb02ef860e7a46d56fef336887e869c490434d: Status 404 returned error can't find the container with id 5ed61a99a29a5fe3572311cbe1bb02ef860e7a46d56fef336887e869c490434d Sep 10 22:11:56.869370 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:11:56.869328 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-578b4977d8-d9snc" event={"ID":"7fb31c55-8407-49ea-9b6f-044e2cf97e12","Type":"ContainerStarted","Data":"5ed61a99a29a5fe3572311cbe1bb02ef860e7a46d56fef336887e869c490434d"} Sep 10 22:12:04.910986 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:04.910948 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-578b4977d8-d9snc" event={"ID":"7fb31c55-8407-49ea-9b6f-044e2cf97e12","Type":"ContainerStarted","Data":"36c2b0d8ccfb22001e8ab7304fb448e03fb02beae8fc2aa77e85d0506b074b0e"} Sep 10 22:12:04.911394 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:04.911036 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-578b4977d8-d9snc" Sep 10 22:12:04.927618 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:04.927572 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-578b4977d8-d9snc" podStartSLOduration=1.449443686 podStartE2EDuration="8.927558185s" totalImagesPullingTime="7.478114499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:12:04.92572726 +0000 UTC m=+426.495217224" watchObservedRunningTime="2026-09-10 22:12:04.927558185 +0000 UTC m=+426.497048151" Sep 10 22:12:06.102006 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:06.101971 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-9b867fd64-b7wgk"] Sep 10 22:12:06.105373 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:06.105357 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9b867fd64-b7wgk" Sep 10 22:12:06.108519 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:06.108496 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 10 22:12:06.109522 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:06.109505 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 22:12:06.110543 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:06.110518 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-97hrc\"" Sep 10 22:12:06.110637 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:06.110568 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 22:12:06.110637 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:06.110610 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 10 22:12:06.122376 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:06.122354 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9b867fd64-b7wgk"] Sep 10 22:12:06.124075 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:06.124049 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5062cdac-ad2e-4b2c-b65c-0a64478e4ec6-tmp\") pod \"kube-auth-proxy-9b867fd64-b7wgk\" (UID: \"5062cdac-ad2e-4b2c-b65c-0a64478e4ec6\") " pod="openshift-ingress/kube-auth-proxy-9b867fd64-b7wgk" Sep 10 22:12:06.124180 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:06.124100 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5062cdac-ad2e-4b2c-b65c-0a64478e4ec6-tls-certs\") pod \"kube-auth-proxy-9b867fd64-b7wgk\" (UID: \"5062cdac-ad2e-4b2c-b65c-0a64478e4ec6\") " pod="openshift-ingress/kube-auth-proxy-9b867fd64-b7wgk" Sep 10 22:12:06.124239 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:06.124209 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq7jt\" (UniqueName: \"kubernetes.io/projected/5062cdac-ad2e-4b2c-b65c-0a64478e4ec6-kube-api-access-mq7jt\") pod \"kube-auth-proxy-9b867fd64-b7wgk\" (UID: \"5062cdac-ad2e-4b2c-b65c-0a64478e4ec6\") " pod="openshift-ingress/kube-auth-proxy-9b867fd64-b7wgk" Sep 10 22:12:06.225305 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:06.225266 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq7jt\" (UniqueName: \"kubernetes.io/projected/5062cdac-ad2e-4b2c-b65c-0a64478e4ec6-kube-api-access-mq7jt\") pod \"kube-auth-proxy-9b867fd64-b7wgk\" (UID: \"5062cdac-ad2e-4b2c-b65c-0a64478e4ec6\") " pod="openshift-ingress/kube-auth-proxy-9b867fd64-b7wgk" Sep 10 22:12:06.225516 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:06.225351 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5062cdac-ad2e-4b2c-b65c-0a64478e4ec6-tmp\") pod \"kube-auth-proxy-9b867fd64-b7wgk\" (UID: \"5062cdac-ad2e-4b2c-b65c-0a64478e4ec6\") " pod="openshift-ingress/kube-auth-proxy-9b867fd64-b7wgk" Sep 10 22:12:06.225516 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:06.225385 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5062cdac-ad2e-4b2c-b65c-0a64478e4ec6-tls-certs\") pod \"kube-auth-proxy-9b867fd64-b7wgk\" (UID: \"5062cdac-ad2e-4b2c-b65c-0a64478e4ec6\") " pod="openshift-ingress/kube-auth-proxy-9b867fd64-b7wgk" Sep 10 22:12:06.227770 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:06.227743 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5062cdac-ad2e-4b2c-b65c-0a64478e4ec6-tmp\") pod \"kube-auth-proxy-9b867fd64-b7wgk\" (UID: \"5062cdac-ad2e-4b2c-b65c-0a64478e4ec6\") " pod="openshift-ingress/kube-auth-proxy-9b867fd64-b7wgk" Sep 10 22:12:06.228014 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:06.227985 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5062cdac-ad2e-4b2c-b65c-0a64478e4ec6-tls-certs\") pod \"kube-auth-proxy-9b867fd64-b7wgk\" (UID: \"5062cdac-ad2e-4b2c-b65c-0a64478e4ec6\") " pod="openshift-ingress/kube-auth-proxy-9b867fd64-b7wgk" Sep 10 22:12:06.233337 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:06.233312 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq7jt\" (UniqueName: \"kubernetes.io/projected/5062cdac-ad2e-4b2c-b65c-0a64478e4ec6-kube-api-access-mq7jt\") pod \"kube-auth-proxy-9b867fd64-b7wgk\" (UID: \"5062cdac-ad2e-4b2c-b65c-0a64478e4ec6\") " pod="openshift-ingress/kube-auth-proxy-9b867fd64-b7wgk" Sep 10 22:12:06.416283 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:06.416202 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9b867fd64-b7wgk" Sep 10 22:12:06.689417 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:06.689386 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9b867fd64-b7wgk"] Sep 10 22:12:06.691968 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:12:06.691939 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5062cdac_ad2e_4b2c_b65c_0a64478e4ec6.slice/crio-ebc81120363b23fdc0ed00c0dbc0012baff3d84f3c1a49c242a393670d90ce82 WatchSource:0}: Error finding container ebc81120363b23fdc0ed00c0dbc0012baff3d84f3c1a49c242a393670d90ce82: Status 404 returned error can't find the container with id ebc81120363b23fdc0ed00c0dbc0012baff3d84f3c1a49c242a393670d90ce82 Sep 10 22:12:06.921940 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:06.921879 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9b867fd64-b7wgk" event={"ID":"5062cdac-ad2e-4b2c-b65c-0a64478e4ec6","Type":"ContainerStarted","Data":"ebc81120363b23fdc0ed00c0dbc0012baff3d84f3c1a49c242a393670d90ce82"} Sep 10 22:12:09.940123 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:09.940088 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9b867fd64-b7wgk" event={"ID":"5062cdac-ad2e-4b2c-b65c-0a64478e4ec6","Type":"ContainerStarted","Data":"70fbdc3f45d1969e49885c1387ba0793a56d08e4710577649081874bb96618de"} Sep 10 22:12:09.959811 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:09.959759 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-9b867fd64-b7wgk" podStartSLOduration=1.2413792479999999 podStartE2EDuration="3.959742641s" totalImagesPullingTime="2.718363393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:12:09.957629506 +0000 UTC m=+431.527119471" watchObservedRunningTime="2026-09-10 22:12:09.959742641 +0000 UTC m=+431.529232607" Sep 10 22:12:15.918342 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:15.918304 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-578b4977d8-d9snc" Sep 10 22:12:26.649350 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:26.649269 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc"] Sep 10 22:12:26.653154 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:26.653134 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc" Sep 10 22:12:26.657438 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:26.657418 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-rg2hf\"" Sep 10 22:12:26.657712 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:26.657696 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 10 22:12:26.662828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:26.662805 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc"] Sep 10 22:12:26.721663 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:26.721636 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvrp9\" (UniqueName: \"kubernetes.io/projected/6dd29f4d-c5ca-43b3-97d4-130855c684b0-kube-api-access-lvrp9\") pod \"llmisvc-controller-manager-777c6bd7d5-tv2jc\" (UID: \"6dd29f4d-c5ca-43b3-97d4-130855c684b0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc" Sep 10 22:12:26.721785 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:26.721680 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6dd29f4d-c5ca-43b3-97d4-130855c684b0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tv2jc\" (UID: \"6dd29f4d-c5ca-43b3-97d4-130855c684b0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc" Sep 10 22:12:26.822284 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:26.822254 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvrp9\" (UniqueName: \"kubernetes.io/projected/6dd29f4d-c5ca-43b3-97d4-130855c684b0-kube-api-access-lvrp9\") pod \"llmisvc-controller-manager-777c6bd7d5-tv2jc\" (UID: \"6dd29f4d-c5ca-43b3-97d4-130855c684b0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc" Sep 10 22:12:26.822428 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:26.822310 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6dd29f4d-c5ca-43b3-97d4-130855c684b0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tv2jc\" (UID: \"6dd29f4d-c5ca-43b3-97d4-130855c684b0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc" Sep 10 22:12:26.824778 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:26.824756 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6dd29f4d-c5ca-43b3-97d4-130855c684b0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tv2jc\" (UID: \"6dd29f4d-c5ca-43b3-97d4-130855c684b0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc" Sep 10 22:12:26.836808 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:26.836789 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvrp9\" (UniqueName: \"kubernetes.io/projected/6dd29f4d-c5ca-43b3-97d4-130855c684b0-kube-api-access-lvrp9\") pod \"llmisvc-controller-manager-777c6bd7d5-tv2jc\" (UID: \"6dd29f4d-c5ca-43b3-97d4-130855c684b0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc" Sep 10 22:12:26.964152 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:26.964073 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc" Sep 10 22:12:27.235251 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:27.235226 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc"] Sep 10 22:12:27.237680 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:12:27.237649 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6dd29f4d_c5ca_43b3_97d4_130855c684b0.slice/crio-44898550b56a9550b30075644fd51c16fc2e649c4e80bade4ad2b29a778a4f62 WatchSource:0}: Error finding container 44898550b56a9550b30075644fd51c16fc2e649c4e80bade4ad2b29a778a4f62: Status 404 returned error can't find the container with id 44898550b56a9550b30075644fd51c16fc2e649c4e80bade4ad2b29a778a4f62 Sep 10 22:12:28.030317 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:28.030283 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc" event={"ID":"6dd29f4d-c5ca-43b3-97d4-130855c684b0","Type":"ContainerStarted","Data":"44898550b56a9550b30075644fd51c16fc2e649c4e80bade4ad2b29a778a4f62"} Sep 10 22:12:31.048454 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:31.048414 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc" event={"ID":"6dd29f4d-c5ca-43b3-97d4-130855c684b0","Type":"ContainerStarted","Data":"08feb0675b1cf9d60057841da8d4c421904ff8103b26ed1fe8a111aab819e4f7"} Sep 10 22:12:31.048942 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:31.048678 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc" Sep 10 22:12:31.072359 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:12:31.072315 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc" podStartSLOduration=2.332697528 podStartE2EDuration="5.072299848s" totalImagesPullingTime="2.73960232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:12:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:12:31.070697802 +0000 UTC m=+452.640187778" watchObservedRunningTime="2026-09-10 22:12:31.072299848 +0000 UTC m=+452.641789813" Sep 10 22:13:02.055075 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:13:02.055043 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc" Sep 10 22:13:55.264335 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:13:55.264252 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7kmv"] Sep 10 22:13:55.267669 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:13:55.267647 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7kmv" Sep 10 22:13:55.270233 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:13:55.270214 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-d2hqd\"" Sep 10 22:13:55.270233 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:13:55.270224 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 10 22:13:55.270368 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:13:55.270212 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 10 22:13:55.282408 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:13:55.282383 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7kmv"] Sep 10 22:13:55.375225 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:13:55.375190 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7csdd\" (UniqueName: \"kubernetes.io/projected/f8226034-0cc3-4303-9d1b-d42162b6c355-kube-api-access-7csdd\") pod \"kuadrant-operator-controller-manager-5769f99d9f-d7kmv\" (UID: \"f8226034-0cc3-4303-9d1b-d42162b6c355\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7kmv" Sep 10 22:13:55.375225 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:13:55.375228 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8226034-0cc3-4303-9d1b-d42162b6c355-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-d7kmv\" (UID: \"f8226034-0cc3-4303-9d1b-d42162b6c355\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7kmv" Sep 10 22:13:55.476378 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:13:55.476346 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7csdd\" (UniqueName: \"kubernetes.io/projected/f8226034-0cc3-4303-9d1b-d42162b6c355-kube-api-access-7csdd\") pod \"kuadrant-operator-controller-manager-5769f99d9f-d7kmv\" (UID: \"f8226034-0cc3-4303-9d1b-d42162b6c355\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7kmv" Sep 10 22:13:55.476378 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:13:55.476382 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8226034-0cc3-4303-9d1b-d42162b6c355-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-d7kmv\" (UID: \"f8226034-0cc3-4303-9d1b-d42162b6c355\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7kmv" Sep 10 22:13:55.476715 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:13:55.476699 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8226034-0cc3-4303-9d1b-d42162b6c355-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-d7kmv\" (UID: \"f8226034-0cc3-4303-9d1b-d42162b6c355\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7kmv" Sep 10 22:13:55.487390 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:13:55.487366 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7csdd\" (UniqueName: \"kubernetes.io/projected/f8226034-0cc3-4303-9d1b-d42162b6c355-kube-api-access-7csdd\") pod \"kuadrant-operator-controller-manager-5769f99d9f-d7kmv\" (UID: \"f8226034-0cc3-4303-9d1b-d42162b6c355\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7kmv" Sep 10 22:13:55.577836 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:13:55.577749 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7kmv" Sep 10 22:13:55.863773 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:13:55.863743 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7kmv"] Sep 10 22:13:55.865377 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:13:55.865349 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8226034_0cc3_4303_9d1b_d42162b6c355.slice/crio-a91a4d96f02313c84a39a73b3d4e2d6fca28eff6f1a59b034ed456c0fd32bb99 WatchSource:0}: Error finding container a91a4d96f02313c84a39a73b3d4e2d6fca28eff6f1a59b034ed456c0fd32bb99: Status 404 returned error can't find the container with id a91a4d96f02313c84a39a73b3d4e2d6fca28eff6f1a59b034ed456c0fd32bb99 Sep 10 22:13:56.440921 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:13:56.440867 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7kmv" event={"ID":"f8226034-0cc3-4303-9d1b-d42162b6c355","Type":"ContainerStarted","Data":"a91a4d96f02313c84a39a73b3d4e2d6fca28eff6f1a59b034ed456c0fd32bb99"} Sep 10 22:14:00.465258 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:00.465219 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7kmv" event={"ID":"f8226034-0cc3-4303-9d1b-d42162b6c355","Type":"ContainerStarted","Data":"f94888076ce29e1a75df83f3e3fa7a687d54059bd7f60b0938df87c906d57f34"} Sep 10 22:14:00.465650 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:00.465290 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7kmv" Sep 10 22:14:00.488789 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:00.488735 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7kmv" podStartSLOduration=1.730605876 podStartE2EDuration="5.488712528s" totalImagesPullingTime="3.758106652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:14:00.487936241 +0000 UTC m=+542.057426205" watchObservedRunningTime="2026-09-10 22:14:00.488712528 +0000 UTC m=+542.058202491" Sep 10 22:14:11.471724 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:11.471692 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7kmv" Sep 10 22:14:32.406669 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:32.406642 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-7mvgd"] Sep 10 22:14:32.409931 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:32.409893 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-7mvgd" Sep 10 22:14:32.412247 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:32.412231 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5kzlj\"" Sep 10 22:14:32.415441 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:32.415412 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-7mvgd"] Sep 10 22:14:32.604207 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:32.604170 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5lsz\" (UniqueName: \"kubernetes.io/projected/f37c0ba1-484a-46bb-bf13-1ccc245153c2-kube-api-access-d5lsz\") pod \"authorino-7bcfd8f9cd-7mvgd\" (UID: \"f37c0ba1-484a-46bb-bf13-1ccc245153c2\") " pod="kuadrant-system/authorino-7bcfd8f9cd-7mvgd" Sep 10 22:14:32.705404 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:32.705315 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5lsz\" (UniqueName: \"kubernetes.io/projected/f37c0ba1-484a-46bb-bf13-1ccc245153c2-kube-api-access-d5lsz\") pod \"authorino-7bcfd8f9cd-7mvgd\" (UID: \"f37c0ba1-484a-46bb-bf13-1ccc245153c2\") " pod="kuadrant-system/authorino-7bcfd8f9cd-7mvgd" Sep 10 22:14:32.714506 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:32.714478 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5lsz\" (UniqueName: \"kubernetes.io/projected/f37c0ba1-484a-46bb-bf13-1ccc245153c2-kube-api-access-d5lsz\") pod \"authorino-7bcfd8f9cd-7mvgd\" (UID: \"f37c0ba1-484a-46bb-bf13-1ccc245153c2\") " pod="kuadrant-system/authorino-7bcfd8f9cd-7mvgd" Sep 10 22:14:32.725265 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:32.725243 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-7mvgd" Sep 10 22:14:32.991937 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:32.991912 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-7mvgd"] Sep 10 22:14:32.994346 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:14:32.994321 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf37c0ba1_484a_46bb_bf13_1ccc245153c2.slice/crio-14ac92dee47b6f4c6dc6be3d4238ce768a7bd1783bf278fce3ca637554f14502 WatchSource:0}: Error finding container 14ac92dee47b6f4c6dc6be3d4238ce768a7bd1783bf278fce3ca637554f14502: Status 404 returned error can't find the container with id 14ac92dee47b6f4c6dc6be3d4238ce768a7bd1783bf278fce3ca637554f14502 Sep 10 22:14:33.616122 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:33.616074 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-7mvgd" event={"ID":"f37c0ba1-484a-46bb-bf13-1ccc245153c2","Type":"ContainerStarted","Data":"14ac92dee47b6f4c6dc6be3d4238ce768a7bd1783bf278fce3ca637554f14502"} Sep 10 22:14:34.621917 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:34.621867 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-7mvgd" event={"ID":"f37c0ba1-484a-46bb-bf13-1ccc245153c2","Type":"ContainerStarted","Data":"a6e92fb78daa16892a2c7c2e5a9da061361971a2b7e226316982b7fbfd16a324"} Sep 10 22:14:34.635724 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:34.635683 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-7mvgd" podStartSLOduration=1.171915491 podStartE2EDuration="2.635667827s" totalImagesPullingTime="1.463752336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:14:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:14:34.635633867 +0000 UTC m=+576.205123831" watchObservedRunningTime="2026-09-10 22:14:34.635667827 +0000 UTC m=+576.205157790" Sep 10 22:14:35.392404 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:35.392364 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-7mvgd"] Sep 10 22:14:36.631916 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:36.631867 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-7mvgd" podUID="f37c0ba1-484a-46bb-bf13-1ccc245153c2" containerName="authorino" containerID="cri-o://a6e92fb78daa16892a2c7c2e5a9da061361971a2b7e226316982b7fbfd16a324" gracePeriod=30 Sep 10 22:14:36.876448 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:36.876427 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-7mvgd" Sep 10 22:14:36.941936 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:36.941872 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5lsz\" (UniqueName: \"kubernetes.io/projected/f37c0ba1-484a-46bb-bf13-1ccc245153c2-kube-api-access-d5lsz\") pod \"f37c0ba1-484a-46bb-bf13-1ccc245153c2\" (UID: \"f37c0ba1-484a-46bb-bf13-1ccc245153c2\") " Sep 10 22:14:36.943754 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:36.943735 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f37c0ba1-484a-46bb-bf13-1ccc245153c2-kube-api-access-d5lsz" (OuterVolumeSpecName: "kube-api-access-d5lsz") pod "f37c0ba1-484a-46bb-bf13-1ccc245153c2" (UID: "f37c0ba1-484a-46bb-bf13-1ccc245153c2"). InnerVolumeSpecName "kube-api-access-d5lsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:14:37.043111 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:37.043087 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5lsz\" (UniqueName: \"kubernetes.io/projected/f37c0ba1-484a-46bb-bf13-1ccc245153c2-kube-api-access-d5lsz\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:14:37.637507 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:37.637473 2564 generic.go:356] "Generic (PLEG): container finished" podID="f37c0ba1-484a-46bb-bf13-1ccc245153c2" containerID="a6e92fb78daa16892a2c7c2e5a9da061361971a2b7e226316982b7fbfd16a324" exitCode=0 Sep 10 22:14:37.637893 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:37.637514 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-7mvgd" Sep 10 22:14:37.637893 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:37.637561 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-7mvgd" event={"ID":"f37c0ba1-484a-46bb-bf13-1ccc245153c2","Type":"ContainerDied","Data":"a6e92fb78daa16892a2c7c2e5a9da061361971a2b7e226316982b7fbfd16a324"} Sep 10 22:14:37.637893 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:37.637605 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-7mvgd" event={"ID":"f37c0ba1-484a-46bb-bf13-1ccc245153c2","Type":"ContainerDied","Data":"14ac92dee47b6f4c6dc6be3d4238ce768a7bd1783bf278fce3ca637554f14502"} Sep 10 22:14:37.637893 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:37.637626 2564 scope.go:117] "RemoveContainer" containerID="a6e92fb78daa16892a2c7c2e5a9da061361971a2b7e226316982b7fbfd16a324" Sep 10 22:14:37.646631 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:37.646614 2564 scope.go:117] "RemoveContainer" containerID="a6e92fb78daa16892a2c7c2e5a9da061361971a2b7e226316982b7fbfd16a324" Sep 10 22:14:37.646894 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:14:37.646871 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e92fb78daa16892a2c7c2e5a9da061361971a2b7e226316982b7fbfd16a324\": container with ID starting with a6e92fb78daa16892a2c7c2e5a9da061361971a2b7e226316982b7fbfd16a324 not found: ID does not exist" containerID="a6e92fb78daa16892a2c7c2e5a9da061361971a2b7e226316982b7fbfd16a324" Sep 10 22:14:37.646963 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:37.646928 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e92fb78daa16892a2c7c2e5a9da061361971a2b7e226316982b7fbfd16a324"} err="failed to get container status \"a6e92fb78daa16892a2c7c2e5a9da061361971a2b7e226316982b7fbfd16a324\": rpc error: code = NotFound desc = could not find container \"a6e92fb78daa16892a2c7c2e5a9da061361971a2b7e226316982b7fbfd16a324\": container with ID starting with a6e92fb78daa16892a2c7c2e5a9da061361971a2b7e226316982b7fbfd16a324 not found: ID does not exist" Sep 10 22:14:37.658889 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:37.658865 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-7mvgd"] Sep 10 22:14:37.664688 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:37.664668 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-7mvgd"] Sep 10 22:14:38.936518 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:38.936491 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f37c0ba1-484a-46bb-bf13-1ccc245153c2" path="/var/lib/kubelet/pods/f37c0ba1-484a-46bb-bf13-1ccc245153c2/volumes" Sep 10 22:14:58.906380 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:58.906351 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-7hrn7"] Sep 10 22:14:58.906812 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:58.906725 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f37c0ba1-484a-46bb-bf13-1ccc245153c2" containerName="authorino" Sep 10 22:14:58.906812 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:58.906736 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f37c0ba1-484a-46bb-bf13-1ccc245153c2" containerName="authorino" Sep 10 22:14:58.906812 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:58.906787 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f37c0ba1-484a-46bb-bf13-1ccc245153c2" containerName="authorino" Sep 10 22:14:58.909739 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:58.909719 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-7hrn7" Sep 10 22:14:58.912132 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:58.912113 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5kzlj\"" Sep 10 22:14:58.915959 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:58.915940 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-7hrn7"] Sep 10 22:14:58.982755 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:58.982730 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-7hrn7"] Sep 10 22:14:58.983059 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:14:58.983035 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-8bn27], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-7hrn7" podUID="3f93f6c8-5386-4a2c-9b27-edc6419a5f8d" Sep 10 22:14:59.008418 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.008391 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-kqtwt"] Sep 10 22:14:59.013065 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.013048 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-kqtwt" Sep 10 22:14:59.015410 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.015392 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 10 22:14:59.021016 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.020998 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-kqtwt"] Sep 10 22:14:59.024652 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.024629 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bn27\" (UniqueName: \"kubernetes.io/projected/3f93f6c8-5386-4a2c-9b27-edc6419a5f8d-kube-api-access-8bn27\") pod \"authorino-557f876f69-7hrn7\" (UID: \"3f93f6c8-5386-4a2c-9b27-edc6419a5f8d\") " pod="kuadrant-system/authorino-557f876f69-7hrn7" Sep 10 22:14:59.125339 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.125311 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bn27\" (UniqueName: \"kubernetes.io/projected/3f93f6c8-5386-4a2c-9b27-edc6419a5f8d-kube-api-access-8bn27\") pod \"authorino-557f876f69-7hrn7\" (UID: \"3f93f6c8-5386-4a2c-9b27-edc6419a5f8d\") " pod="kuadrant-system/authorino-557f876f69-7hrn7" Sep 10 22:14:59.125444 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.125376 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6986c66c-5e61-4958-bd39-722941d1f750-tls-cert\") pod \"authorino-6d4dfc649f-kqtwt\" (UID: \"6986c66c-5e61-4958-bd39-722941d1f750\") " pod="kuadrant-system/authorino-6d4dfc649f-kqtwt" Sep 10 22:14:59.125444 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.125407 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7gj4\" (UniqueName: \"kubernetes.io/projected/6986c66c-5e61-4958-bd39-722941d1f750-kube-api-access-g7gj4\") pod \"authorino-6d4dfc649f-kqtwt\" (UID: \"6986c66c-5e61-4958-bd39-722941d1f750\") " pod="kuadrant-system/authorino-6d4dfc649f-kqtwt" Sep 10 22:14:59.132820 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.132799 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bn27\" (UniqueName: \"kubernetes.io/projected/3f93f6c8-5386-4a2c-9b27-edc6419a5f8d-kube-api-access-8bn27\") pod \"authorino-557f876f69-7hrn7\" (UID: \"3f93f6c8-5386-4a2c-9b27-edc6419a5f8d\") " pod="kuadrant-system/authorino-557f876f69-7hrn7" Sep 10 22:14:59.144700 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.144679 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-kqtwt"] Sep 10 22:14:59.144911 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:14:59.144885 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-g7gj4 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-kqtwt" podUID="6986c66c-5e61-4958-bd39-722941d1f750" Sep 10 22:14:59.225978 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.225924 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6986c66c-5e61-4958-bd39-722941d1f750-tls-cert\") pod \"authorino-6d4dfc649f-kqtwt\" (UID: \"6986c66c-5e61-4958-bd39-722941d1f750\") " pod="kuadrant-system/authorino-6d4dfc649f-kqtwt" Sep 10 22:14:59.225978 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.225956 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7gj4\" (UniqueName: \"kubernetes.io/projected/6986c66c-5e61-4958-bd39-722941d1f750-kube-api-access-g7gj4\") pod \"authorino-6d4dfc649f-kqtwt\" (UID: \"6986c66c-5e61-4958-bd39-722941d1f750\") " pod="kuadrant-system/authorino-6d4dfc649f-kqtwt" Sep 10 22:14:59.228182 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.228161 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6986c66c-5e61-4958-bd39-722941d1f750-tls-cert\") pod \"authorino-6d4dfc649f-kqtwt\" (UID: \"6986c66c-5e61-4958-bd39-722941d1f750\") " pod="kuadrant-system/authorino-6d4dfc649f-kqtwt" Sep 10 22:14:59.232990 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.232966 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7gj4\" (UniqueName: \"kubernetes.io/projected/6986c66c-5e61-4958-bd39-722941d1f750-kube-api-access-g7gj4\") pod \"authorino-6d4dfc649f-kqtwt\" (UID: \"6986c66c-5e61-4958-bd39-722941d1f750\") " pod="kuadrant-system/authorino-6d4dfc649f-kqtwt" Sep 10 22:14:59.752632 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.752601 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-kqtwt" Sep 10 22:14:59.752784 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.752770 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-7hrn7" Sep 10 22:14:59.765546 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.765523 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-kqtwt" Sep 10 22:14:59.768232 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.768214 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-7hrn7" Sep 10 22:14:59.829846 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.829824 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6986c66c-5e61-4958-bd39-722941d1f750-tls-cert\") pod \"6986c66c-5e61-4958-bd39-722941d1f750\" (UID: \"6986c66c-5e61-4958-bd39-722941d1f750\") " Sep 10 22:14:59.829945 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.829871 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bn27\" (UniqueName: \"kubernetes.io/projected/3f93f6c8-5386-4a2c-9b27-edc6419a5f8d-kube-api-access-8bn27\") pod \"3f93f6c8-5386-4a2c-9b27-edc6419a5f8d\" (UID: \"3f93f6c8-5386-4a2c-9b27-edc6419a5f8d\") " Sep 10 22:14:59.829945 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.829895 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7gj4\" (UniqueName: \"kubernetes.io/projected/6986c66c-5e61-4958-bd39-722941d1f750-kube-api-access-g7gj4\") pod \"6986c66c-5e61-4958-bd39-722941d1f750\" (UID: \"6986c66c-5e61-4958-bd39-722941d1f750\") " Sep 10 22:14:59.831873 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.831850 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6986c66c-5e61-4958-bd39-722941d1f750-kube-api-access-g7gj4" (OuterVolumeSpecName: "kube-api-access-g7gj4") pod "6986c66c-5e61-4958-bd39-722941d1f750" (UID: "6986c66c-5e61-4958-bd39-722941d1f750"). InnerVolumeSpecName "kube-api-access-g7gj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:14:59.831973 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.831878 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f93f6c8-5386-4a2c-9b27-edc6419a5f8d-kube-api-access-8bn27" (OuterVolumeSpecName: "kube-api-access-8bn27") pod "3f93f6c8-5386-4a2c-9b27-edc6419a5f8d" (UID: "3f93f6c8-5386-4a2c-9b27-edc6419a5f8d"). InnerVolumeSpecName "kube-api-access-8bn27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:14:59.831973 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.831948 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6986c66c-5e61-4958-bd39-722941d1f750-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "6986c66c-5e61-4958-bd39-722941d1f750" (UID: "6986c66c-5e61-4958-bd39-722941d1f750"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:14:59.930982 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.930960 2564 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6986c66c-5e61-4958-bd39-722941d1f750-tls-cert\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:14:59.930982 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.930980 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bn27\" (UniqueName: \"kubernetes.io/projected/3f93f6c8-5386-4a2c-9b27-edc6419a5f8d-kube-api-access-8bn27\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:14:59.931289 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:14:59.930989 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7gj4\" (UniqueName: \"kubernetes.io/projected/6986c66c-5e61-4958-bd39-722941d1f750-kube-api-access-g7gj4\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:15:00.757416 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:00.757386 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-7hrn7" Sep 10 22:15:00.757584 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:00.757387 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-kqtwt" Sep 10 22:15:00.786409 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:00.786382 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-kqtwt"] Sep 10 22:15:00.789582 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:00.789561 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-kqtwt"] Sep 10 22:15:00.808034 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:00.808014 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-7hrn7"] Sep 10 22:15:00.809928 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:00.809911 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-7hrn7"] Sep 10 22:15:00.936596 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:00.936568 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f93f6c8-5386-4a2c-9b27-edc6419a5f8d" path="/var/lib/kubelet/pods/3f93f6c8-5386-4a2c-9b27-edc6419a5f8d/volumes" Sep 10 22:15:00.936886 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:00.936804 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6986c66c-5e61-4958-bd39-722941d1f750" path="/var/lib/kubelet/pods/6986c66c-5e61-4958-bd39-722941d1f750/volumes" Sep 10 22:15:06.187113 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.187080 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-766fc89467-qm49l"] Sep 10 22:15:06.190690 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.190670 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766fc89467-qm49l" Sep 10 22:15:06.193808 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.193783 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zr6dm\"" Sep 10 22:15:06.194076 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.194057 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 22:15:06.194145 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.194089 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 22:15:06.194868 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.194850 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 22:15:06.207619 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.207550 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-766fc89467-qm49l"] Sep 10 22:15:06.284285 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.284258 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f531ede-a030-4157-b431-93ab39f37cb0-webhook-cert\") pod \"maas-controller-766fc89467-qm49l\" (UID: \"9f531ede-a030-4157-b431-93ab39f37cb0\") " pod="opendatahub/maas-controller-766fc89467-qm49l" Sep 10 22:15:06.284389 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.284295 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrjw7\" (UniqueName: \"kubernetes.io/projected/9f531ede-a030-4157-b431-93ab39f37cb0-kube-api-access-mrjw7\") pod \"maas-controller-766fc89467-qm49l\" (UID: \"9f531ede-a030-4157-b431-93ab39f37cb0\") " pod="opendatahub/maas-controller-766fc89467-qm49l" Sep 10 22:15:06.284389 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.284312 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f531ede-a030-4157-b431-93ab39f37cb0-metrics-certs\") pod \"maas-controller-766fc89467-qm49l\" (UID: \"9f531ede-a030-4157-b431-93ab39f37cb0\") " pod="opendatahub/maas-controller-766fc89467-qm49l" Sep 10 22:15:06.364457 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.364436 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b6f6f558b-jc7ct"] Sep 10 22:15:06.367735 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.367718 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b6f6f558b-jc7ct" Sep 10 22:15:06.378074 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.378054 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b6f6f558b-jc7ct"] Sep 10 22:15:06.385341 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.385315 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f531ede-a030-4157-b431-93ab39f37cb0-webhook-cert\") pod \"maas-controller-766fc89467-qm49l\" (UID: \"9f531ede-a030-4157-b431-93ab39f37cb0\") " pod="opendatahub/maas-controller-766fc89467-qm49l" Sep 10 22:15:06.385426 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.385367 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrjw7\" (UniqueName: \"kubernetes.io/projected/9f531ede-a030-4157-b431-93ab39f37cb0-kube-api-access-mrjw7\") pod \"maas-controller-766fc89467-qm49l\" (UID: \"9f531ede-a030-4157-b431-93ab39f37cb0\") " pod="opendatahub/maas-controller-766fc89467-qm49l" Sep 10 22:15:06.385426 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.385399 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f531ede-a030-4157-b431-93ab39f37cb0-metrics-certs\") pod \"maas-controller-766fc89467-qm49l\" (UID: \"9f531ede-a030-4157-b431-93ab39f37cb0\") " pod="opendatahub/maas-controller-766fc89467-qm49l" Sep 10 22:15:06.385503 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:15:06.385450 2564 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 10 22:15:06.385503 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:15:06.385501 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f531ede-a030-4157-b431-93ab39f37cb0-webhook-cert podName:9f531ede-a030-4157-b431-93ab39f37cb0 nodeName:}" failed. No retries permitted until 2026-09-10 22:15:06.885486784 +0000 UTC m=+608.454976731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/9f531ede-a030-4157-b431-93ab39f37cb0-webhook-cert") pod "maas-controller-766fc89467-qm49l" (UID: "9f531ede-a030-4157-b431-93ab39f37cb0") : secret "maas-controller-webhook-cert" not found Sep 10 22:15:06.388165 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.388143 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f531ede-a030-4157-b431-93ab39f37cb0-metrics-certs\") pod \"maas-controller-766fc89467-qm49l\" (UID: \"9f531ede-a030-4157-b431-93ab39f37cb0\") " pod="opendatahub/maas-controller-766fc89467-qm49l" Sep 10 22:15:06.395661 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.395638 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrjw7\" (UniqueName: \"kubernetes.io/projected/9f531ede-a030-4157-b431-93ab39f37cb0-kube-api-access-mrjw7\") pod \"maas-controller-766fc89467-qm49l\" (UID: \"9f531ede-a030-4157-b431-93ab39f37cb0\") " pod="opendatahub/maas-controller-766fc89467-qm49l" Sep 10 22:15:06.486309 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.486258 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0-metrics-certs\") pod \"maas-controller-5b6f6f558b-jc7ct\" (UID: \"27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0\") " pod="opendatahub/maas-controller-5b6f6f558b-jc7ct" Sep 10 22:15:06.486309 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.486284 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0-webhook-cert\") pod \"maas-controller-5b6f6f558b-jc7ct\" (UID: \"27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0\") " pod="opendatahub/maas-controller-5b6f6f558b-jc7ct" Sep 10 22:15:06.486431 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.486316 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99c7w\" (UniqueName: \"kubernetes.io/projected/27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0-kube-api-access-99c7w\") pod \"maas-controller-5b6f6f558b-jc7ct\" (UID: \"27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0\") " pod="opendatahub/maas-controller-5b6f6f558b-jc7ct" Sep 10 22:15:06.587604 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.587581 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0-webhook-cert\") pod \"maas-controller-5b6f6f558b-jc7ct\" (UID: \"27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0\") " pod="opendatahub/maas-controller-5b6f6f558b-jc7ct" Sep 10 22:15:06.587689 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.587622 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99c7w\" (UniqueName: \"kubernetes.io/projected/27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0-kube-api-access-99c7w\") pod \"maas-controller-5b6f6f558b-jc7ct\" (UID: \"27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0\") " pod="opendatahub/maas-controller-5b6f6f558b-jc7ct" Sep 10 22:15:06.587728 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.587691 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0-metrics-certs\") pod \"maas-controller-5b6f6f558b-jc7ct\" (UID: \"27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0\") " pod="opendatahub/maas-controller-5b6f6f558b-jc7ct" Sep 10 22:15:06.587728 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:15:06.587716 2564 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 10 22:15:06.587789 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:15:06.587780 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0-webhook-cert podName:27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0 nodeName:}" failed. No retries permitted until 2026-09-10 22:15:07.08776523 +0000 UTC m=+608.657255172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0-webhook-cert") pod "maas-controller-5b6f6f558b-jc7ct" (UID: "27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0") : secret "maas-controller-webhook-cert" not found Sep 10 22:15:06.589874 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.589856 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0-metrics-certs\") pod \"maas-controller-5b6f6f558b-jc7ct\" (UID: \"27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0\") " pod="opendatahub/maas-controller-5b6f6f558b-jc7ct" Sep 10 22:15:06.595538 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.595515 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99c7w\" (UniqueName: \"kubernetes.io/projected/27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0-kube-api-access-99c7w\") pod \"maas-controller-5b6f6f558b-jc7ct\" (UID: \"27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0\") " pod="opendatahub/maas-controller-5b6f6f558b-jc7ct" Sep 10 22:15:06.890657 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.890624 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f531ede-a030-4157-b431-93ab39f37cb0-webhook-cert\") pod \"maas-controller-766fc89467-qm49l\" (UID: \"9f531ede-a030-4157-b431-93ab39f37cb0\") " pod="opendatahub/maas-controller-766fc89467-qm49l" Sep 10 22:15:06.893111 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:06.893079 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f531ede-a030-4157-b431-93ab39f37cb0-webhook-cert\") pod \"maas-controller-766fc89467-qm49l\" (UID: \"9f531ede-a030-4157-b431-93ab39f37cb0\") " pod="opendatahub/maas-controller-766fc89467-qm49l" Sep 10 22:15:07.092670 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:07.092636 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0-webhook-cert\") pod \"maas-controller-5b6f6f558b-jc7ct\" (UID: \"27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0\") " pod="opendatahub/maas-controller-5b6f6f558b-jc7ct" Sep 10 22:15:07.094954 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:07.094931 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0-webhook-cert\") pod \"maas-controller-5b6f6f558b-jc7ct\" (UID: \"27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0\") " pod="opendatahub/maas-controller-5b6f6f558b-jc7ct" Sep 10 22:15:07.100815 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:07.100799 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766fc89467-qm49l" Sep 10 22:15:07.277919 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:07.277860 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b6f6f558b-jc7ct" Sep 10 22:15:07.367580 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:07.367558 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-766fc89467-qm49l"] Sep 10 22:15:07.368940 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:15:07.368892 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f531ede_a030_4157_b431_93ab39f37cb0.slice/crio-a821f6ac279f5d1e671fe37ae8d621f818d0cacf7286fdcd7e107e993a70663d WatchSource:0}: Error finding container a821f6ac279f5d1e671fe37ae8d621f818d0cacf7286fdcd7e107e993a70663d: Status 404 returned error can't find the container with id a821f6ac279f5d1e671fe37ae8d621f818d0cacf7286fdcd7e107e993a70663d Sep 10 22:15:07.549541 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:07.549516 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b6f6f558b-jc7ct"] Sep 10 22:15:07.551143 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:15:07.551109 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f3569c_9ada_48fd_9e3d_bc5c7bbcabc0.slice/crio-fe18a37fe06c4a5169a892eb66c0a1df9097d8ead98a2620e23656396e3b4b37 WatchSource:0}: Error finding container fe18a37fe06c4a5169a892eb66c0a1df9097d8ead98a2620e23656396e3b4b37: Status 404 returned error can't find the container with id fe18a37fe06c4a5169a892eb66c0a1df9097d8ead98a2620e23656396e3b4b37 Sep 10 22:15:07.790562 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:07.790527 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b6f6f558b-jc7ct" event={"ID":"27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0","Type":"ContainerStarted","Data":"fe18a37fe06c4a5169a892eb66c0a1df9097d8ead98a2620e23656396e3b4b37"} Sep 10 22:15:07.791736 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:07.791716 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766fc89467-qm49l" event={"ID":"9f531ede-a030-4157-b431-93ab39f37cb0","Type":"ContainerStarted","Data":"a821f6ac279f5d1e671fe37ae8d621f818d0cacf7286fdcd7e107e993a70663d"} Sep 10 22:15:10.813420 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:10.813376 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b6f6f558b-jc7ct" event={"ID":"27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0","Type":"ContainerStarted","Data":"38fe75a95ad594703011c4a447dfd36aad0221266668d0738eac2974e88f1392"} Sep 10 22:15:10.813826 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:10.813481 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b6f6f558b-jc7ct" Sep 10 22:15:10.814951 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:10.814929 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766fc89467-qm49l" event={"ID":"9f531ede-a030-4157-b431-93ab39f37cb0","Type":"ContainerStarted","Data":"945d4714ad856cb388d1df03976030eeee600bd837140afc0ce62bae3cb2f611"} Sep 10 22:15:10.815065 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:10.814978 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-766fc89467-qm49l" Sep 10 22:15:10.830851 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:10.830815 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b6f6f558b-jc7ct" podStartSLOduration=2.440293285 podStartE2EDuration="4.83080334s" totalImagesPullingTime="2.390510055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:15:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:15:10.82934766 +0000 UTC m=+612.398837621" watchObservedRunningTime="2026-09-10 22:15:10.83080334 +0000 UTC m=+612.400293295" Sep 10 22:15:10.847352 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:10.847308 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-766fc89467-qm49l" podStartSLOduration=2.284091587 podStartE2EDuration="4.847295297s" totalImagesPullingTime="2.56320371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:15:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:15:10.846249003 +0000 UTC m=+612.415738968" watchObservedRunningTime="2026-09-10 22:15:10.847295297 +0000 UTC m=+612.416785260" Sep 10 22:15:21.822152 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:21.822073 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-766fc89467-qm49l" Sep 10 22:15:21.822590 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:21.822240 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b6f6f558b-jc7ct" Sep 10 22:15:21.895426 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:21.895395 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-766fc89467-qm49l"] Sep 10 22:15:21.895643 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:21.895596 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-766fc89467-qm49l" podUID="9f531ede-a030-4157-b431-93ab39f37cb0" containerName="manager" containerID="cri-o://945d4714ad856cb388d1df03976030eeee600bd837140afc0ce62bae3cb2f611" gracePeriod=30 Sep 10 22:15:22.139737 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:22.139710 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766fc89467-qm49l" Sep 10 22:15:22.227432 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:22.227397 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f531ede-a030-4157-b431-93ab39f37cb0-webhook-cert\") pod \"9f531ede-a030-4157-b431-93ab39f37cb0\" (UID: \"9f531ede-a030-4157-b431-93ab39f37cb0\") " Sep 10 22:15:22.227619 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:22.227451 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f531ede-a030-4157-b431-93ab39f37cb0-metrics-certs\") pod \"9f531ede-a030-4157-b431-93ab39f37cb0\" (UID: \"9f531ede-a030-4157-b431-93ab39f37cb0\") " Sep 10 22:15:22.227619 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:22.227485 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrjw7\" (UniqueName: \"kubernetes.io/projected/9f531ede-a030-4157-b431-93ab39f37cb0-kube-api-access-mrjw7\") pod \"9f531ede-a030-4157-b431-93ab39f37cb0\" (UID: \"9f531ede-a030-4157-b431-93ab39f37cb0\") " Sep 10 22:15:22.229740 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:22.229713 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f531ede-a030-4157-b431-93ab39f37cb0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9f531ede-a030-4157-b431-93ab39f37cb0" (UID: "9f531ede-a030-4157-b431-93ab39f37cb0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:15:22.229845 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:22.229792 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f531ede-a030-4157-b431-93ab39f37cb0-kube-api-access-mrjw7" (OuterVolumeSpecName: "kube-api-access-mrjw7") pod "9f531ede-a030-4157-b431-93ab39f37cb0" (UID: "9f531ede-a030-4157-b431-93ab39f37cb0"). InnerVolumeSpecName "kube-api-access-mrjw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:15:22.229845 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:22.229831 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f531ede-a030-4157-b431-93ab39f37cb0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9f531ede-a030-4157-b431-93ab39f37cb0" (UID: "9f531ede-a030-4157-b431-93ab39f37cb0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:15:22.328178 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:22.328146 2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f531ede-a030-4157-b431-93ab39f37cb0-webhook-cert\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:15:22.328178 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:22.328174 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f531ede-a030-4157-b431-93ab39f37cb0-metrics-certs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:15:22.328178 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:22.328184 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrjw7\" (UniqueName: \"kubernetes.io/projected/9f531ede-a030-4157-b431-93ab39f37cb0-kube-api-access-mrjw7\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:15:22.873199 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:22.873161 2564 generic.go:356] "Generic (PLEG): container finished" podID="9f531ede-a030-4157-b431-93ab39f37cb0" containerID="945d4714ad856cb388d1df03976030eeee600bd837140afc0ce62bae3cb2f611" exitCode=0 Sep 10 22:15:22.873670 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:22.873209 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766fc89467-qm49l" Sep 10 22:15:22.873670 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:22.873253 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766fc89467-qm49l" event={"ID":"9f531ede-a030-4157-b431-93ab39f37cb0","Type":"ContainerDied","Data":"945d4714ad856cb388d1df03976030eeee600bd837140afc0ce62bae3cb2f611"} Sep 10 22:15:22.873670 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:22.873319 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766fc89467-qm49l" event={"ID":"9f531ede-a030-4157-b431-93ab39f37cb0","Type":"ContainerDied","Data":"a821f6ac279f5d1e671fe37ae8d621f818d0cacf7286fdcd7e107e993a70663d"} Sep 10 22:15:22.873670 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:22.873352 2564 scope.go:117] "RemoveContainer" containerID="945d4714ad856cb388d1df03976030eeee600bd837140afc0ce62bae3cb2f611" Sep 10 22:15:22.882379 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:22.882361 2564 scope.go:117] "RemoveContainer" containerID="945d4714ad856cb388d1df03976030eeee600bd837140afc0ce62bae3cb2f611" Sep 10 22:15:22.882634 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:15:22.882617 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"945d4714ad856cb388d1df03976030eeee600bd837140afc0ce62bae3cb2f611\": container with ID starting with 945d4714ad856cb388d1df03976030eeee600bd837140afc0ce62bae3cb2f611 not found: ID does not exist" containerID="945d4714ad856cb388d1df03976030eeee600bd837140afc0ce62bae3cb2f611" Sep 10 22:15:22.882708 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:22.882639 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945d4714ad856cb388d1df03976030eeee600bd837140afc0ce62bae3cb2f611"} err="failed to get container status \"945d4714ad856cb388d1df03976030eeee600bd837140afc0ce62bae3cb2f611\": rpc error: code = NotFound desc = could not find container \"945d4714ad856cb388d1df03976030eeee600bd837140afc0ce62bae3cb2f611\": container with ID starting with 945d4714ad856cb388d1df03976030eeee600bd837140afc0ce62bae3cb2f611 not found: ID does not exist" Sep 10 22:15:22.894477 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:22.894452 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-766fc89467-qm49l"] Sep 10 22:15:22.897564 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:22.897542 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-766fc89467-qm49l"] Sep 10 22:15:22.936707 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:22.936678 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f531ede-a030-4157-b431-93ab39f37cb0" path="/var/lib/kubelet/pods/9f531ede-a030-4157-b431-93ab39f37cb0/volumes" Sep 10 22:15:46.078608 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.078578 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-554988677c-lxsxm"] Sep 10 22:15:46.079019 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.078936 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f531ede-a030-4157-b431-93ab39f37cb0" containerName="manager" Sep 10 22:15:46.079019 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.078946 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f531ede-a030-4157-b431-93ab39f37cb0" containerName="manager" Sep 10 22:15:46.079019 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.079018 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f531ede-a030-4157-b431-93ab39f37cb0" containerName="manager" Sep 10 22:15:46.085246 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.085230 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-554988677c-lxsxm" Sep 10 22:15:46.088311 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.088286 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 22:15:46.088311 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.088290 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 10 22:15:46.089494 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.089477 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zvqzg\"" Sep 10 22:15:46.089818 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.089797 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 10 22:15:46.089927 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.089822 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 10 22:15:46.094389 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.094359 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-554988677c-lxsxm"] Sep 10 22:15:46.147640 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.147614 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zstx9\" (UniqueName: \"kubernetes.io/projected/11db8a20-d065-41c8-8fb7-3bc09a778ba3-kube-api-access-zstx9\") pod \"maas-api-554988677c-lxsxm\" (UID: \"11db8a20-d065-41c8-8fb7-3bc09a778ba3\") " pod="odh-ai-gateway-infra/maas-api-554988677c-lxsxm" Sep 10 22:15:46.147770 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.147677 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11db8a20-d065-41c8-8fb7-3bc09a778ba3-openshift-service-ca\") pod \"maas-api-554988677c-lxsxm\" (UID: \"11db8a20-d065-41c8-8fb7-3bc09a778ba3\") " pod="odh-ai-gateway-infra/maas-api-554988677c-lxsxm" Sep 10 22:15:46.147770 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.147703 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11db8a20-d065-41c8-8fb7-3bc09a778ba3-metrics-certs\") pod \"maas-api-554988677c-lxsxm\" (UID: \"11db8a20-d065-41c8-8fb7-3bc09a778ba3\") " pod="odh-ai-gateway-infra/maas-api-554988677c-lxsxm" Sep 10 22:15:46.147877 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.147800 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11db8a20-d065-41c8-8fb7-3bc09a778ba3-maas-api-tls\") pod \"maas-api-554988677c-lxsxm\" (UID: \"11db8a20-d065-41c8-8fb7-3bc09a778ba3\") " pod="odh-ai-gateway-infra/maas-api-554988677c-lxsxm" Sep 10 22:15:46.248279 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.248259 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11db8a20-d065-41c8-8fb7-3bc09a778ba3-openshift-service-ca\") pod \"maas-api-554988677c-lxsxm\" (UID: \"11db8a20-d065-41c8-8fb7-3bc09a778ba3\") " pod="odh-ai-gateway-infra/maas-api-554988677c-lxsxm" Sep 10 22:15:46.248397 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.248286 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11db8a20-d065-41c8-8fb7-3bc09a778ba3-metrics-certs\") pod \"maas-api-554988677c-lxsxm\" (UID: \"11db8a20-d065-41c8-8fb7-3bc09a778ba3\") " pod="odh-ai-gateway-infra/maas-api-554988677c-lxsxm" Sep 10 22:15:46.248397 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.248331 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11db8a20-d065-41c8-8fb7-3bc09a778ba3-maas-api-tls\") pod \"maas-api-554988677c-lxsxm\" (UID: \"11db8a20-d065-41c8-8fb7-3bc09a778ba3\") " pod="odh-ai-gateway-infra/maas-api-554988677c-lxsxm" Sep 10 22:15:46.248397 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.248356 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zstx9\" (UniqueName: \"kubernetes.io/projected/11db8a20-d065-41c8-8fb7-3bc09a778ba3-kube-api-access-zstx9\") pod \"maas-api-554988677c-lxsxm\" (UID: \"11db8a20-d065-41c8-8fb7-3bc09a778ba3\") " pod="odh-ai-gateway-infra/maas-api-554988677c-lxsxm" Sep 10 22:15:46.248966 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.248946 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11db8a20-d065-41c8-8fb7-3bc09a778ba3-openshift-service-ca\") pod \"maas-api-554988677c-lxsxm\" (UID: \"11db8a20-d065-41c8-8fb7-3bc09a778ba3\") " pod="odh-ai-gateway-infra/maas-api-554988677c-lxsxm" Sep 10 22:15:46.250624 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.250603 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11db8a20-d065-41c8-8fb7-3bc09a778ba3-metrics-certs\") pod \"maas-api-554988677c-lxsxm\" (UID: \"11db8a20-d065-41c8-8fb7-3bc09a778ba3\") " pod="odh-ai-gateway-infra/maas-api-554988677c-lxsxm" Sep 10 22:15:46.250726 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.250703 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11db8a20-d065-41c8-8fb7-3bc09a778ba3-maas-api-tls\") pod \"maas-api-554988677c-lxsxm\" (UID: \"11db8a20-d065-41c8-8fb7-3bc09a778ba3\") " pod="odh-ai-gateway-infra/maas-api-554988677c-lxsxm" Sep 10 22:15:46.262756 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.258197 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zstx9\" (UniqueName: \"kubernetes.io/projected/11db8a20-d065-41c8-8fb7-3bc09a778ba3-kube-api-access-zstx9\") pod \"maas-api-554988677c-lxsxm\" (UID: \"11db8a20-d065-41c8-8fb7-3bc09a778ba3\") " pod="odh-ai-gateway-infra/maas-api-554988677c-lxsxm" Sep 10 22:15:46.397082 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.397048 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-554988677c-lxsxm" Sep 10 22:15:46.677815 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.677695 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-554988677c-lxsxm"] Sep 10 22:15:46.680572 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:15:46.680544 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11db8a20_d065_41c8_8fb7_3bc09a778ba3.slice/crio-6abacaf3765f6fe1829f3b6ddea52d805272c4aa48a384bc660567deb48794e4 WatchSource:0}: Error finding container 6abacaf3765f6fe1829f3b6ddea52d805272c4aa48a384bc660567deb48794e4: Status 404 returned error can't find the container with id 6abacaf3765f6fe1829f3b6ddea52d805272c4aa48a384bc660567deb48794e4 Sep 10 22:15:46.993081 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:46.992999 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-554988677c-lxsxm" event={"ID":"11db8a20-d065-41c8-8fb7-3bc09a778ba3","Type":"ContainerStarted","Data":"6abacaf3765f6fe1829f3b6ddea52d805272c4aa48a384bc660567deb48794e4"} Sep 10 22:15:49.005190 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:49.005154 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-554988677c-lxsxm" event={"ID":"11db8a20-d065-41c8-8fb7-3bc09a778ba3","Type":"ContainerStarted","Data":"c72053253f1e0be307197c2eb4b90c19160279bb316188f17298a9d43fdf3a17"} Sep 10 22:15:49.005643 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:49.005216 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-554988677c-lxsxm" Sep 10 22:15:49.022501 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:49.022459 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-554988677c-lxsxm" podStartSLOduration=1.29733051 podStartE2EDuration="3.022444119s" totalImagesPullingTime="1.725113609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:15:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:15:49.02108035 +0000 UTC m=+650.590570314" watchObservedRunningTime="2026-09-10 22:15:49.022444119 +0000 UTC m=+650.591934082" Sep 10 22:15:55.014746 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:15:55.014715 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-554988677c-lxsxm" Sep 10 22:16:07.308008 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.307972 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n"] Sep 10 22:16:07.313312 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.313287 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" Sep 10 22:16:07.316983 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.316963 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 22:16:07.317107 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.316969 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 10 22:16:07.317208 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.317189 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-v5j2d\"" Sep 10 22:16:07.317270 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.317258 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 22:16:07.320886 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.320460 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n"] Sep 10 22:16:07.329941 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.329882 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73cadc3a-00f0-41b0-9e8d-9a0839c6800e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kt72n\" (UID: \"73cadc3a-00f0-41b0-9e8d-9a0839c6800e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" Sep 10 22:16:07.330074 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.329960 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73cadc3a-00f0-41b0-9e8d-9a0839c6800e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kt72n\" (UID: \"73cadc3a-00f0-41b0-9e8d-9a0839c6800e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" Sep 10 22:16:07.330074 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.330003 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz9lm\" (UniqueName: \"kubernetes.io/projected/73cadc3a-00f0-41b0-9e8d-9a0839c6800e-kube-api-access-cz9lm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kt72n\" (UID: \"73cadc3a-00f0-41b0-9e8d-9a0839c6800e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" Sep 10 22:16:07.330205 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.330119 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73cadc3a-00f0-41b0-9e8d-9a0839c6800e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kt72n\" (UID: \"73cadc3a-00f0-41b0-9e8d-9a0839c6800e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" Sep 10 22:16:07.330205 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.330163 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73cadc3a-00f0-41b0-9e8d-9a0839c6800e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kt72n\" (UID: \"73cadc3a-00f0-41b0-9e8d-9a0839c6800e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" Sep 10 22:16:07.330205 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.330190 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73cadc3a-00f0-41b0-9e8d-9a0839c6800e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kt72n\" (UID: \"73cadc3a-00f0-41b0-9e8d-9a0839c6800e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" Sep 10 22:16:07.431628 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.431596 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73cadc3a-00f0-41b0-9e8d-9a0839c6800e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kt72n\" (UID: \"73cadc3a-00f0-41b0-9e8d-9a0839c6800e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" Sep 10 22:16:07.431828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.431637 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73cadc3a-00f0-41b0-9e8d-9a0839c6800e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kt72n\" (UID: \"73cadc3a-00f0-41b0-9e8d-9a0839c6800e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" Sep 10 22:16:07.431828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.431657 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73cadc3a-00f0-41b0-9e8d-9a0839c6800e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kt72n\" (UID: \"73cadc3a-00f0-41b0-9e8d-9a0839c6800e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" Sep 10 22:16:07.431828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.431692 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73cadc3a-00f0-41b0-9e8d-9a0839c6800e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kt72n\" (UID: \"73cadc3a-00f0-41b0-9e8d-9a0839c6800e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" Sep 10 22:16:07.431828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.431713 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73cadc3a-00f0-41b0-9e8d-9a0839c6800e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kt72n\" (UID: \"73cadc3a-00f0-41b0-9e8d-9a0839c6800e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" Sep 10 22:16:07.431828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.431747 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz9lm\" (UniqueName: \"kubernetes.io/projected/73cadc3a-00f0-41b0-9e8d-9a0839c6800e-kube-api-access-cz9lm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kt72n\" (UID: \"73cadc3a-00f0-41b0-9e8d-9a0839c6800e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" Sep 10 22:16:07.432153 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.432096 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73cadc3a-00f0-41b0-9e8d-9a0839c6800e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kt72n\" (UID: \"73cadc3a-00f0-41b0-9e8d-9a0839c6800e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" Sep 10 22:16:07.432153 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.432131 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73cadc3a-00f0-41b0-9e8d-9a0839c6800e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kt72n\" (UID: \"73cadc3a-00f0-41b0-9e8d-9a0839c6800e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" Sep 10 22:16:07.432229 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.432201 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73cadc3a-00f0-41b0-9e8d-9a0839c6800e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kt72n\" (UID: \"73cadc3a-00f0-41b0-9e8d-9a0839c6800e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" Sep 10 22:16:07.434022 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.433999 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73cadc3a-00f0-41b0-9e8d-9a0839c6800e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kt72n\" (UID: \"73cadc3a-00f0-41b0-9e8d-9a0839c6800e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" Sep 10 22:16:07.434166 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.434150 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73cadc3a-00f0-41b0-9e8d-9a0839c6800e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kt72n\" (UID: \"73cadc3a-00f0-41b0-9e8d-9a0839c6800e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" Sep 10 22:16:07.439733 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.439714 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz9lm\" (UniqueName: \"kubernetes.io/projected/73cadc3a-00f0-41b0-9e8d-9a0839c6800e-kube-api-access-cz9lm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kt72n\" (UID: \"73cadc3a-00f0-41b0-9e8d-9a0839c6800e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" Sep 10 22:16:07.625965 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.625932 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" Sep 10 22:16:07.896248 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:07.896220 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n"] Sep 10 22:16:07.898228 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:16:07.898199 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73cadc3a_00f0_41b0_9e8d_9a0839c6800e.slice/crio-bc71fe9a744e0bdc61758823a74891e39b8a593bf38ddaa6a5901417eeb3bf37 WatchSource:0}: Error finding container bc71fe9a744e0bdc61758823a74891e39b8a593bf38ddaa6a5901417eeb3bf37: Status 404 returned error can't find the container with id bc71fe9a744e0bdc61758823a74891e39b8a593bf38ddaa6a5901417eeb3bf37 Sep 10 22:16:08.095351 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:08.095317 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" event={"ID":"73cadc3a-00f0-41b0-9e8d-9a0839c6800e","Type":"ContainerStarted","Data":"bc71fe9a744e0bdc61758823a74891e39b8a593bf38ddaa6a5901417eeb3bf37"} Sep 10 22:16:09.100579 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:09.100545 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" event={"ID":"73cadc3a-00f0-41b0-9e8d-9a0839c6800e","Type":"ContainerStarted","Data":"18266affd7bc8b417e0887b127adb1a8a3e73e08bf3be26ea46742c32f906936"} Sep 10 22:16:09.118605 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:09.118557 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" podStartSLOduration=1.074570555 podStartE2EDuration="2.118544009s" totalImagesPullingTime="1.043973454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:16:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:16:09.1161492 +0000 UTC m=+670.685639162" watchObservedRunningTime="2026-09-10 22:16:09.118544009 +0000 UTC m=+670.688033973" Sep 10 22:16:17.626862 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:17.626809 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" Sep 10 22:16:17.627290 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:17.626885 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" Sep 10 22:16:17.639503 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:17.639473 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" Sep 10 22:16:18.155135 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:18.155108 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kt72n" Sep 10 22:16:50.599572 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:50.599543 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj"] Sep 10 22:16:50.605792 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:50.605771 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" Sep 10 22:16:50.608641 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:50.608615 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 10 22:16:50.615912 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:50.615879 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj"] Sep 10 22:16:50.634757 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:50.634716 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc904b9f-ab74-48db-9212-324873622c3d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj\" (UID: \"fc904b9f-ab74-48db-9212-324873622c3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" Sep 10 22:16:50.634930 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:50.634777 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6647g\" (UniqueName: \"kubernetes.io/projected/fc904b9f-ab74-48db-9212-324873622c3d-kube-api-access-6647g\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj\" (UID: \"fc904b9f-ab74-48db-9212-324873622c3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" Sep 10 22:16:50.634930 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:50.634810 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc904b9f-ab74-48db-9212-324873622c3d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj\" (UID: \"fc904b9f-ab74-48db-9212-324873622c3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" Sep 10 22:16:50.634930 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:50.634915 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc904b9f-ab74-48db-9212-324873622c3d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj\" (UID: \"fc904b9f-ab74-48db-9212-324873622c3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" Sep 10 22:16:50.635058 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:50.634949 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc904b9f-ab74-48db-9212-324873622c3d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj\" (UID: \"fc904b9f-ab74-48db-9212-324873622c3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" Sep 10 22:16:50.635058 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:50.635021 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc904b9f-ab74-48db-9212-324873622c3d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj\" (UID: \"fc904b9f-ab74-48db-9212-324873622c3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" Sep 10 22:16:50.735802 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:50.735768 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc904b9f-ab74-48db-9212-324873622c3d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj\" (UID: \"fc904b9f-ab74-48db-9212-324873622c3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" Sep 10 22:16:50.735802 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:50.735801 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc904b9f-ab74-48db-9212-324873622c3d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj\" (UID: \"fc904b9f-ab74-48db-9212-324873622c3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" Sep 10 22:16:50.736065 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:50.735852 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc904b9f-ab74-48db-9212-324873622c3d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj\" (UID: \"fc904b9f-ab74-48db-9212-324873622c3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" Sep 10 22:16:50.736065 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:50.735958 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc904b9f-ab74-48db-9212-324873622c3d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj\" (UID: \"fc904b9f-ab74-48db-9212-324873622c3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" Sep 10 22:16:50.736065 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:50.736003 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6647g\" (UniqueName: \"kubernetes.io/projected/fc904b9f-ab74-48db-9212-324873622c3d-kube-api-access-6647g\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj\" (UID: \"fc904b9f-ab74-48db-9212-324873622c3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" Sep 10 22:16:50.736065 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:50.736048 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc904b9f-ab74-48db-9212-324873622c3d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj\" (UID: \"fc904b9f-ab74-48db-9212-324873622c3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" Sep 10 22:16:50.736342 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:50.736230 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc904b9f-ab74-48db-9212-324873622c3d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj\" (UID: \"fc904b9f-ab74-48db-9212-324873622c3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" Sep 10 22:16:50.736342 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:50.736293 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc904b9f-ab74-48db-9212-324873622c3d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj\" (UID: \"fc904b9f-ab74-48db-9212-324873622c3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" Sep 10 22:16:50.736448 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:50.736389 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc904b9f-ab74-48db-9212-324873622c3d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj\" (UID: \"fc904b9f-ab74-48db-9212-324873622c3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" Sep 10 22:16:50.738144 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:50.738120 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc904b9f-ab74-48db-9212-324873622c3d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj\" (UID: \"fc904b9f-ab74-48db-9212-324873622c3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" Sep 10 22:16:50.738335 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:50.738318 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc904b9f-ab74-48db-9212-324873622c3d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj\" (UID: \"fc904b9f-ab74-48db-9212-324873622c3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" Sep 10 22:16:50.743754 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:50.743728 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6647g\" (UniqueName: \"kubernetes.io/projected/fc904b9f-ab74-48db-9212-324873622c3d-kube-api-access-6647g\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj\" (UID: \"fc904b9f-ab74-48db-9212-324873622c3d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" Sep 10 22:16:50.918219 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:50.918180 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" Sep 10 22:16:51.188682 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:51.188659 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj"] Sep 10 22:16:51.190774 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:16:51.190738 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc904b9f_ab74_48db_9212_324873622c3d.slice/crio-32b01daf922fd6d14789fe4cfd585428af518abbd4f32d5e78bdc6413a97502e WatchSource:0}: Error finding container 32b01daf922fd6d14789fe4cfd585428af518abbd4f32d5e78bdc6413a97502e: Status 404 returned error can't find the container with id 32b01daf922fd6d14789fe4cfd585428af518abbd4f32d5e78bdc6413a97502e Sep 10 22:16:51.192498 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:51.192484 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 22:16:51.308152 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:51.308115 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" event={"ID":"fc904b9f-ab74-48db-9212-324873622c3d","Type":"ContainerStarted","Data":"32b01daf922fd6d14789fe4cfd585428af518abbd4f32d5e78bdc6413a97502e"} Sep 10 22:16:52.314311 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:52.314277 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" event={"ID":"fc904b9f-ab74-48db-9212-324873622c3d","Type":"ContainerStarted","Data":"9d43877bdd16873e3d1780f1ebaae9032b3e99f0218214b2f1808fad2cb06fac"} Sep 10 22:16:52.331173 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:16:52.331125 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" podStartSLOduration=2.145193919 podStartE2EDuration="2.331108942s" totalImagesPullingTime="185.915023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:16:52.329364944 +0000 UTC m=+713.898854909" watchObservedRunningTime="2026-09-10 22:16:52.331108942 +0000 UTC m=+713.900598906" Sep 10 22:17:00.919300 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:17:00.919261 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" Sep 10 22:17:00.919690 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:17:00.919323 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" Sep 10 22:17:00.936975 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:17:00.936950 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" Sep 10 22:17:01.372949 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:17:01.372918 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj" Sep 10 22:18:26.577075 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:18:26.577003 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5474b4d86d-hqp85"] Sep 10 22:18:26.580449 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:18:26.580430 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5474b4d86d-hqp85" Sep 10 22:18:26.584192 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:18:26.584162 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5kzlj\"" Sep 10 22:18:26.584519 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:18:26.584502 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 10 22:18:26.586852 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:18:26.586832 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5474b4d86d-hqp85"] Sep 10 22:18:26.735853 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:18:26.735822 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqbws\" (UniqueName: \"kubernetes.io/projected/4f72c97e-e212-4128-9251-35fe4ca5fdb3-kube-api-access-vqbws\") pod \"authorino-5474b4d86d-hqp85\" (UID: \"4f72c97e-e212-4128-9251-35fe4ca5fdb3\") " pod="kuadrant-system/authorino-5474b4d86d-hqp85" Sep 10 22:18:26.736045 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:18:26.735886 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4f72c97e-e212-4128-9251-35fe4ca5fdb3-tls-cert\") pod \"authorino-5474b4d86d-hqp85\" (UID: \"4f72c97e-e212-4128-9251-35fe4ca5fdb3\") " pod="kuadrant-system/authorino-5474b4d86d-hqp85" Sep 10 22:18:26.837351 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:18:26.837285 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqbws\" (UniqueName: \"kubernetes.io/projected/4f72c97e-e212-4128-9251-35fe4ca5fdb3-kube-api-access-vqbws\") pod \"authorino-5474b4d86d-hqp85\" (UID: \"4f72c97e-e212-4128-9251-35fe4ca5fdb3\") " pod="kuadrant-system/authorino-5474b4d86d-hqp85" Sep 10 22:18:26.837351 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:18:26.837333 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4f72c97e-e212-4128-9251-35fe4ca5fdb3-tls-cert\") pod \"authorino-5474b4d86d-hqp85\" (UID: \"4f72c97e-e212-4128-9251-35fe4ca5fdb3\") " pod="kuadrant-system/authorino-5474b4d86d-hqp85" Sep 10 22:18:26.839619 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:18:26.839599 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4f72c97e-e212-4128-9251-35fe4ca5fdb3-tls-cert\") pod \"authorino-5474b4d86d-hqp85\" (UID: \"4f72c97e-e212-4128-9251-35fe4ca5fdb3\") " pod="kuadrant-system/authorino-5474b4d86d-hqp85" Sep 10 22:18:26.845324 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:18:26.845306 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqbws\" (UniqueName: \"kubernetes.io/projected/4f72c97e-e212-4128-9251-35fe4ca5fdb3-kube-api-access-vqbws\") pod \"authorino-5474b4d86d-hqp85\" (UID: \"4f72c97e-e212-4128-9251-35fe4ca5fdb3\") " pod="kuadrant-system/authorino-5474b4d86d-hqp85" Sep 10 22:18:26.897063 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:18:26.897028 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5474b4d86d-hqp85" Sep 10 22:18:27.167087 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:18:27.167063 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5474b4d86d-hqp85"] Sep 10 22:18:27.168711 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:18:27.168684 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f72c97e_e212_4128_9251_35fe4ca5fdb3.slice/crio-e366daefd5c40fb734da3f1193cc9931be4ffc88af84283011f86bf05a3b5f65 WatchSource:0}: Error finding container e366daefd5c40fb734da3f1193cc9931be4ffc88af84283011f86bf05a3b5f65: Status 404 returned error can't find the container with id e366daefd5c40fb734da3f1193cc9931be4ffc88af84283011f86bf05a3b5f65 Sep 10 22:18:27.775871 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:18:27.775835 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5474b4d86d-hqp85" event={"ID":"4f72c97e-e212-4128-9251-35fe4ca5fdb3","Type":"ContainerStarted","Data":"e366daefd5c40fb734da3f1193cc9931be4ffc88af84283011f86bf05a3b5f65"} Sep 10 22:18:45.869694 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:18:45.869656 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5474b4d86d-hqp85" event={"ID":"4f72c97e-e212-4128-9251-35fe4ca5fdb3","Type":"ContainerStarted","Data":"f6cbcf046b37d50feae892691f00e95e8acbe4a6b24da4b70dc541ace090bb25"} Sep 10 22:18:45.891149 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:18:45.891098 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5474b4d86d-hqp85" podStartSLOduration=1.748029125 podStartE2EDuration="19.891081671s" totalImagesPullingTime="18.143052546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:18:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:18:45.889537695 +0000 UTC m=+827.459027676" watchObservedRunningTime="2026-09-10 22:18:45.891081671 +0000 UTC m=+827.460571634" Sep 10 22:19:07.805371 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:07.805342 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g"] Sep 10 22:19:07.808998 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:07.808977 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g" Sep 10 22:19:07.811225 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:07.811205 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-669e2b-serving-cert\"" Sep 10 22:19:07.820102 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:07.820075 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g"] Sep 10 22:19:07.893769 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:07.893736 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n56nb\" (UniqueName: \"kubernetes.io/projected/371a2114-dc48-4bfb-82c2-a44550f9cf67-kube-api-access-n56nb\") pod \"maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g\" (UID: \"371a2114-dc48-4bfb-82c2-a44550f9cf67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g" Sep 10 22:19:07.893942 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:07.893780 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/371a2114-dc48-4bfb-82c2-a44550f9cf67-openshift-service-ca\") pod \"maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g\" (UID: \"371a2114-dc48-4bfb-82c2-a44550f9cf67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g" Sep 10 22:19:07.893942 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:07.893849 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/371a2114-dc48-4bfb-82c2-a44550f9cf67-maas-api-tls\") pod \"maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g\" (UID: \"371a2114-dc48-4bfb-82c2-a44550f9cf67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g" Sep 10 22:19:07.894041 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:07.893966 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371a2114-dc48-4bfb-82c2-a44550f9cf67-metrics-certs\") pod \"maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g\" (UID: \"371a2114-dc48-4bfb-82c2-a44550f9cf67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g" Sep 10 22:19:07.994493 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:07.994466 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/371a2114-dc48-4bfb-82c2-a44550f9cf67-maas-api-tls\") pod \"maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g\" (UID: \"371a2114-dc48-4bfb-82c2-a44550f9cf67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g" Sep 10 22:19:07.994636 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:07.994535 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371a2114-dc48-4bfb-82c2-a44550f9cf67-metrics-certs\") pod \"maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g\" (UID: \"371a2114-dc48-4bfb-82c2-a44550f9cf67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g" Sep 10 22:19:07.994636 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:07.994569 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n56nb\" (UniqueName: \"kubernetes.io/projected/371a2114-dc48-4bfb-82c2-a44550f9cf67-kube-api-access-n56nb\") pod \"maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g\" (UID: \"371a2114-dc48-4bfb-82c2-a44550f9cf67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g" Sep 10 22:19:07.994636 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:07.994591 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/371a2114-dc48-4bfb-82c2-a44550f9cf67-openshift-service-ca\") pod \"maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g\" (UID: \"371a2114-dc48-4bfb-82c2-a44550f9cf67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g" Sep 10 22:19:07.995198 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:07.995181 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/371a2114-dc48-4bfb-82c2-a44550f9cf67-openshift-service-ca\") pod \"maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g\" (UID: \"371a2114-dc48-4bfb-82c2-a44550f9cf67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g" Sep 10 22:19:07.996991 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:07.996974 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/371a2114-dc48-4bfb-82c2-a44550f9cf67-maas-api-tls\") pod \"maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g\" (UID: \"371a2114-dc48-4bfb-82c2-a44550f9cf67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g" Sep 10 22:19:07.997059 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:07.997031 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371a2114-dc48-4bfb-82c2-a44550f9cf67-metrics-certs\") pod \"maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g\" (UID: \"371a2114-dc48-4bfb-82c2-a44550f9cf67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g" Sep 10 22:19:08.004512 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:08.004487 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n56nb\" (UniqueName: \"kubernetes.io/projected/371a2114-dc48-4bfb-82c2-a44550f9cf67-kube-api-access-n56nb\") pod \"maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g\" (UID: \"371a2114-dc48-4bfb-82c2-a44550f9cf67\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g" Sep 10 22:19:08.120767 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:08.120735 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g" Sep 10 22:19:08.401123 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:08.401101 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g"] Sep 10 22:19:08.403490 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:19:08.403462 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371a2114_dc48_4bfb_82c2_a44550f9cf67.slice/crio-b33be3755c7a3e6bf11440d315a141e8f51d9f3545187133c7f2d680f44b87f6 WatchSource:0}: Error finding container b33be3755c7a3e6bf11440d315a141e8f51d9f3545187133c7f2d680f44b87f6: Status 404 returned error can't find the container with id b33be3755c7a3e6bf11440d315a141e8f51d9f3545187133c7f2d680f44b87f6 Sep 10 22:19:08.978551 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:08.978520 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g" event={"ID":"371a2114-dc48-4bfb-82c2-a44550f9cf67","Type":"ContainerStarted","Data":"ce3c541039cbe389410727fcf0a09d38ea352d4600ce5e10a886a91ccac66277"} Sep 10 22:19:08.978885 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:08.978566 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g" event={"ID":"371a2114-dc48-4bfb-82c2-a44550f9cf67","Type":"ContainerStarted","Data":"b33be3755c7a3e6bf11440d315a141e8f51d9f3545187133c7f2d680f44b87f6"} Sep 10 22:19:08.978885 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:08.978636 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g" Sep 10 22:19:08.994518 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:08.994477 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g" podStartSLOduration=1.539827308 podStartE2EDuration="1.994463414s" totalImagesPullingTime="454.636106ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:19:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:19:08.99362164 +0000 UTC m=+850.563111605" watchObservedRunningTime="2026-09-10 22:19:08.994463414 +0000 UTC m=+850.563953378" Sep 10 22:19:14.592765 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:14.592733 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh"] Sep 10 22:19:14.597318 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:14.597290 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh" Sep 10 22:19:14.599576 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:14.599549 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-938f94d8-serving-cert\"" Sep 10 22:19:14.607634 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:14.607601 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh"] Sep 10 22:19:14.755189 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:14.755152 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88h8m\" (UniqueName: \"kubernetes.io/projected/4298b2c8-c852-4bc9-b72c-a686041af6f2-kube-api-access-88h8m\") pod \"maas-api-e2e-ait-938f94d8-64b4b79997-447lh\" (UID: \"4298b2c8-c852-4bc9-b72c-a686041af6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh" Sep 10 22:19:14.755189 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:14.755194 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4298b2c8-c852-4bc9-b72c-a686041af6f2-openshift-service-ca\") pod \"maas-api-e2e-ait-938f94d8-64b4b79997-447lh\" (UID: \"4298b2c8-c852-4bc9-b72c-a686041af6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh" Sep 10 22:19:14.755418 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:14.755221 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4298b2c8-c852-4bc9-b72c-a686041af6f2-maas-api-tls\") pod \"maas-api-e2e-ait-938f94d8-64b4b79997-447lh\" (UID: \"4298b2c8-c852-4bc9-b72c-a686041af6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh" Sep 10 22:19:14.755418 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:14.755358 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4298b2c8-c852-4bc9-b72c-a686041af6f2-metrics-certs\") pod \"maas-api-e2e-ait-938f94d8-64b4b79997-447lh\" (UID: \"4298b2c8-c852-4bc9-b72c-a686041af6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh" Sep 10 22:19:14.855981 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:14.855937 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88h8m\" (UniqueName: \"kubernetes.io/projected/4298b2c8-c852-4bc9-b72c-a686041af6f2-kube-api-access-88h8m\") pod \"maas-api-e2e-ait-938f94d8-64b4b79997-447lh\" (UID: \"4298b2c8-c852-4bc9-b72c-a686041af6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh" Sep 10 22:19:14.855981 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:14.855983 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4298b2c8-c852-4bc9-b72c-a686041af6f2-openshift-service-ca\") pod \"maas-api-e2e-ait-938f94d8-64b4b79997-447lh\" (UID: \"4298b2c8-c852-4bc9-b72c-a686041af6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh" Sep 10 22:19:14.856262 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:14.856020 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4298b2c8-c852-4bc9-b72c-a686041af6f2-maas-api-tls\") pod \"maas-api-e2e-ait-938f94d8-64b4b79997-447lh\" (UID: \"4298b2c8-c852-4bc9-b72c-a686041af6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh" Sep 10 22:19:14.856262 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:14.856081 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4298b2c8-c852-4bc9-b72c-a686041af6f2-metrics-certs\") pod \"maas-api-e2e-ait-938f94d8-64b4b79997-447lh\" (UID: \"4298b2c8-c852-4bc9-b72c-a686041af6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh" Sep 10 22:19:14.856262 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:19:14.856156 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-serving-cert: secret "maas-api-e2e-ait-938f94d8-serving-cert" not found Sep 10 22:19:14.856262 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:19:14.856227 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4298b2c8-c852-4bc9-b72c-a686041af6f2-maas-api-tls podName:4298b2c8-c852-4bc9-b72c-a686041af6f2 nodeName:}" failed. No retries permitted until 2026-09-10 22:19:15.356208048 +0000 UTC m=+856.925697996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4298b2c8-c852-4bc9-b72c-a686041af6f2-maas-api-tls") pod "maas-api-e2e-ait-938f94d8-64b4b79997-447lh" (UID: "4298b2c8-c852-4bc9-b72c-a686041af6f2") : secret "maas-api-e2e-ait-938f94d8-serving-cert" not found Sep 10 22:19:14.856720 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:14.856699 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4298b2c8-c852-4bc9-b72c-a686041af6f2-openshift-service-ca\") pod \"maas-api-e2e-ait-938f94d8-64b4b79997-447lh\" (UID: \"4298b2c8-c852-4bc9-b72c-a686041af6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh" Sep 10 22:19:14.858603 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:14.858584 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4298b2c8-c852-4bc9-b72c-a686041af6f2-metrics-certs\") pod \"maas-api-e2e-ait-938f94d8-64b4b79997-447lh\" (UID: \"4298b2c8-c852-4bc9-b72c-a686041af6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh" Sep 10 22:19:14.864813 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:14.864792 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88h8m\" (UniqueName: \"kubernetes.io/projected/4298b2c8-c852-4bc9-b72c-a686041af6f2-kube-api-access-88h8m\") pod \"maas-api-e2e-ait-938f94d8-64b4b79997-447lh\" (UID: \"4298b2c8-c852-4bc9-b72c-a686041af6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh" Sep 10 22:19:14.988944 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:14.988892 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g" Sep 10 22:19:15.360534 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:15.360500 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4298b2c8-c852-4bc9-b72c-a686041af6f2-maas-api-tls\") pod \"maas-api-e2e-ait-938f94d8-64b4b79997-447lh\" (UID: \"4298b2c8-c852-4bc9-b72c-a686041af6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh" Sep 10 22:19:15.363127 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:15.363092 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4298b2c8-c852-4bc9-b72c-a686041af6f2-maas-api-tls\") pod \"maas-api-e2e-ait-938f94d8-64b4b79997-447lh\" (UID: \"4298b2c8-c852-4bc9-b72c-a686041af6f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh" Sep 10 22:19:15.510740 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:15.510700 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh" Sep 10 22:19:15.798933 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:15.798895 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh"] Sep 10 22:19:15.801019 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:19:15.800991 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4298b2c8_c852_4bc9_b72c_a686041af6f2.slice/crio-476393bd5ba4f40c70c3d8bb8c4ac5f43775f9a90579a5009ec219130a02791b WatchSource:0}: Error finding container 476393bd5ba4f40c70c3d8bb8c4ac5f43775f9a90579a5009ec219130a02791b: Status 404 returned error can't find the container with id 476393bd5ba4f40c70c3d8bb8c4ac5f43775f9a90579a5009ec219130a02791b Sep 10 22:19:16.015970 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:16.015932 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh" event={"ID":"4298b2c8-c852-4bc9-b72c-a686041af6f2","Type":"ContainerStarted","Data":"476393bd5ba4f40c70c3d8bb8c4ac5f43775f9a90579a5009ec219130a02791b"} Sep 10 22:19:16.914365 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:16.914335 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb"] Sep 10 22:19:16.923326 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:16.923300 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" Sep 10 22:19:16.926810 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:16.926787 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-44dc54-serving-cert\"" Sep 10 22:19:16.927674 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:16.927614 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb"] Sep 10 22:19:17.022525 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:17.022483 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh" event={"ID":"4298b2c8-c852-4bc9-b72c-a686041af6f2","Type":"ContainerStarted","Data":"e7ddf2ff5d1b6ec5f7d15dbc2a164c5cd29afef92dc8c0fb3ac893ccb50b48e0"} Sep 10 22:19:17.022741 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:17.022548 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh" Sep 10 22:19:17.042430 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:17.042361 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh" podStartSLOduration=2.745233009 podStartE2EDuration="3.042339037s" totalImagesPullingTime="297.106028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:19:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:19:17.041709002 +0000 UTC m=+858.611198978" watchObservedRunningTime="2026-09-10 22:19:17.042339037 +0000 UTC m=+858.611829002" Sep 10 22:19:17.077756 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:17.077712 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d11d98c9-655d-46f4-b5ce-51a850d64a25-metrics-certs\") pod \"maas-api-e2e-autores-44dc54-6b45894bc7-c74pb\" (UID: \"d11d98c9-655d-46f4-b5ce-51a850d64a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" Sep 10 22:19:17.077982 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:17.077772 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d11d98c9-655d-46f4-b5ce-51a850d64a25-openshift-service-ca\") pod \"maas-api-e2e-autores-44dc54-6b45894bc7-c74pb\" (UID: \"d11d98c9-655d-46f4-b5ce-51a850d64a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" Sep 10 22:19:17.077982 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:17.077845 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzs8d\" (UniqueName: \"kubernetes.io/projected/d11d98c9-655d-46f4-b5ce-51a850d64a25-kube-api-access-jzs8d\") pod \"maas-api-e2e-autores-44dc54-6b45894bc7-c74pb\" (UID: \"d11d98c9-655d-46f4-b5ce-51a850d64a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" Sep 10 22:19:17.078104 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:17.077973 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d11d98c9-655d-46f4-b5ce-51a850d64a25-maas-api-tls\") pod \"maas-api-e2e-autores-44dc54-6b45894bc7-c74pb\" (UID: \"d11d98c9-655d-46f4-b5ce-51a850d64a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" Sep 10 22:19:17.179166 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:17.179073 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d11d98c9-655d-46f4-b5ce-51a850d64a25-maas-api-tls\") pod \"maas-api-e2e-autores-44dc54-6b45894bc7-c74pb\" (UID: \"d11d98c9-655d-46f4-b5ce-51a850d64a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" Sep 10 22:19:17.179166 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:17.179147 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d11d98c9-655d-46f4-b5ce-51a850d64a25-metrics-certs\") pod \"maas-api-e2e-autores-44dc54-6b45894bc7-c74pb\" (UID: \"d11d98c9-655d-46f4-b5ce-51a850d64a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" Sep 10 22:19:17.179403 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:17.179204 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d11d98c9-655d-46f4-b5ce-51a850d64a25-openshift-service-ca\") pod \"maas-api-e2e-autores-44dc54-6b45894bc7-c74pb\" (UID: \"d11d98c9-655d-46f4-b5ce-51a850d64a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" Sep 10 22:19:17.179403 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:19:17.179226 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-serving-cert: secret "maas-api-e2e-autores-44dc54-serving-cert" not found Sep 10 22:19:17.179403 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:17.179270 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzs8d\" (UniqueName: \"kubernetes.io/projected/d11d98c9-655d-46f4-b5ce-51a850d64a25-kube-api-access-jzs8d\") pod \"maas-api-e2e-autores-44dc54-6b45894bc7-c74pb\" (UID: \"d11d98c9-655d-46f4-b5ce-51a850d64a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" Sep 10 22:19:17.179403 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:19:17.179310 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11d98c9-655d-46f4-b5ce-51a850d64a25-maas-api-tls podName:d11d98c9-655d-46f4-b5ce-51a850d64a25 nodeName:}" failed. No retries permitted until 2026-09-10 22:19:17.679287695 +0000 UTC m=+859.248777657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d11d98c9-655d-46f4-b5ce-51a850d64a25-maas-api-tls") pod "maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" (UID: "d11d98c9-655d-46f4-b5ce-51a850d64a25") : secret "maas-api-e2e-autores-44dc54-serving-cert" not found Sep 10 22:19:17.180095 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:17.180069 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d11d98c9-655d-46f4-b5ce-51a850d64a25-openshift-service-ca\") pod \"maas-api-e2e-autores-44dc54-6b45894bc7-c74pb\" (UID: \"d11d98c9-655d-46f4-b5ce-51a850d64a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" Sep 10 22:19:17.181887 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:17.181862 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d11d98c9-655d-46f4-b5ce-51a850d64a25-metrics-certs\") pod \"maas-api-e2e-autores-44dc54-6b45894bc7-c74pb\" (UID: \"d11d98c9-655d-46f4-b5ce-51a850d64a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" Sep 10 22:19:17.195444 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:17.195415 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzs8d\" (UniqueName: \"kubernetes.io/projected/d11d98c9-655d-46f4-b5ce-51a850d64a25-kube-api-access-jzs8d\") pod \"maas-api-e2e-autores-44dc54-6b45894bc7-c74pb\" (UID: \"d11d98c9-655d-46f4-b5ce-51a850d64a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" Sep 10 22:19:17.684426 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:17.684380 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d11d98c9-655d-46f4-b5ce-51a850d64a25-maas-api-tls\") pod \"maas-api-e2e-autores-44dc54-6b45894bc7-c74pb\" (UID: \"d11d98c9-655d-46f4-b5ce-51a850d64a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" Sep 10 22:19:17.686883 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:17.686855 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d11d98c9-655d-46f4-b5ce-51a850d64a25-maas-api-tls\") pod \"maas-api-e2e-autores-44dc54-6b45894bc7-c74pb\" (UID: \"d11d98c9-655d-46f4-b5ce-51a850d64a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" Sep 10 22:19:17.836663 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:17.836622 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" Sep 10 22:19:18.114645 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:18.114615 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb"] Sep 10 22:19:18.116632 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:19:18.116608 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11d98c9_655d_46f4_b5ce_51a850d64a25.slice/crio-cd0cb27ae4df3736d6abec70bc85fbd32cadfa30bdcf9e7d9925c2d107cb9498 WatchSource:0}: Error finding container cd0cb27ae4df3736d6abec70bc85fbd32cadfa30bdcf9e7d9925c2d107cb9498: Status 404 returned error can't find the container with id cd0cb27ae4df3736d6abec70bc85fbd32cadfa30bdcf9e7d9925c2d107cb9498 Sep 10 22:19:19.032778 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:19.032743 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" event={"ID":"d11d98c9-655d-46f4-b5ce-51a850d64a25","Type":"ContainerStarted","Data":"c0669f1559eab505fc649728fcdeece92f6abb07120dddb278c05fdbc3de65fd"} Sep 10 22:19:19.032979 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:19.032802 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" event={"ID":"d11d98c9-655d-46f4-b5ce-51a850d64a25","Type":"ContainerStarted","Data":"cd0cb27ae4df3736d6abec70bc85fbd32cadfa30bdcf9e7d9925c2d107cb9498"} Sep 10 22:19:19.032979 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:19.032935 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" Sep 10 22:19:19.052699 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:19.052634 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" podStartSLOduration=2.692169319 podStartE2EDuration="3.052610175s" totalImagesPullingTime="360.440856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:19:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:19:19.050339188 +0000 UTC m=+860.619829192" watchObservedRunningTime="2026-09-10 22:19:19.052610175 +0000 UTC m=+860.622100139" Sep 10 22:19:23.032838 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:23.032812 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh" Sep 10 22:19:25.043760 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:25.043730 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" Sep 10 22:19:36.305228 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:36.305188 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5"] Sep 10 22:19:36.310978 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:36.309875 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" Sep 10 22:19:36.313704 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:36.313465 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c3af05-serving-cert\"" Sep 10 22:19:36.321285 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:36.321257 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5"] Sep 10 22:19:36.361103 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:36.361071 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-metrics-certs\") pod \"maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5\" (UID: \"82cb99cc-1e95-43ee-8209-f9113ed0e5ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" Sep 10 22:19:36.361279 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:36.361173 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-maas-api-tls\") pod \"maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5\" (UID: \"82cb99cc-1e95-43ee-8209-f9113ed0e5ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" Sep 10 22:19:36.361279 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:36.361226 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5\" (UID: \"82cb99cc-1e95-43ee-8209-f9113ed0e5ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" Sep 10 22:19:36.361279 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:36.361245 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77s2l\" (UniqueName: \"kubernetes.io/projected/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-kube-api-access-77s2l\") pod \"maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5\" (UID: \"82cb99cc-1e95-43ee-8209-f9113ed0e5ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" Sep 10 22:19:36.462616 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:36.462523 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-metrics-certs\") pod \"maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5\" (UID: \"82cb99cc-1e95-43ee-8209-f9113ed0e5ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" Sep 10 22:19:36.462767 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:36.462638 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-maas-api-tls\") pod \"maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5\" (UID: \"82cb99cc-1e95-43ee-8209-f9113ed0e5ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" Sep 10 22:19:36.462767 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:36.462671 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5\" (UID: \"82cb99cc-1e95-43ee-8209-f9113ed0e5ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" Sep 10 22:19:36.462767 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:36.462694 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77s2l\" (UniqueName: \"kubernetes.io/projected/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-kube-api-access-77s2l\") pod \"maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5\" (UID: \"82cb99cc-1e95-43ee-8209-f9113ed0e5ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" Sep 10 22:19:36.462892 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:19:36.462772 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-serving-cert: secret "maas-api-e2e-inf-b-c3af05-serving-cert" not found Sep 10 22:19:36.462892 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:19:36.462862 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-maas-api-tls podName:82cb99cc-1e95-43ee-8209-f9113ed0e5ea nodeName:}" failed. No retries permitted until 2026-09-10 22:19:36.962840324 +0000 UTC m=+878.532330283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-maas-api-tls") pod "maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" (UID: "82cb99cc-1e95-43ee-8209-f9113ed0e5ea") : secret "maas-api-e2e-inf-b-c3af05-serving-cert" not found Sep 10 22:19:36.463352 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:36.463334 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5\" (UID: \"82cb99cc-1e95-43ee-8209-f9113ed0e5ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" Sep 10 22:19:36.464996 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:36.464973 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-metrics-certs\") pod \"maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5\" (UID: \"82cb99cc-1e95-43ee-8209-f9113ed0e5ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" Sep 10 22:19:36.473158 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:36.473135 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77s2l\" (UniqueName: \"kubernetes.io/projected/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-kube-api-access-77s2l\") pod \"maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5\" (UID: \"82cb99cc-1e95-43ee-8209-f9113ed0e5ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" Sep 10 22:19:36.969054 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:36.969010 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-maas-api-tls\") pod \"maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5\" (UID: \"82cb99cc-1e95-43ee-8209-f9113ed0e5ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" Sep 10 22:19:36.971333 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:36.971307 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-maas-api-tls\") pod \"maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5\" (UID: \"82cb99cc-1e95-43ee-8209-f9113ed0e5ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" Sep 10 22:19:37.226618 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:37.226535 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" Sep 10 22:19:37.517181 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:37.517155 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5"] Sep 10 22:19:37.519475 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:19:37.519441 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82cb99cc_1e95_43ee_8209_f9113ed0e5ea.slice/crio-9a92b8773b704f5d506f77cc4b4f704cd61a6c1ca68cba9743772731a6db8a31 WatchSource:0}: Error finding container 9a92b8773b704f5d506f77cc4b4f704cd61a6c1ca68cba9743772731a6db8a31: Status 404 returned error can't find the container with id 9a92b8773b704f5d506f77cc4b4f704cd61a6c1ca68cba9743772731a6db8a31 Sep 10 22:19:38.132551 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:38.132518 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" event={"ID":"82cb99cc-1e95-43ee-8209-f9113ed0e5ea","Type":"ContainerStarted","Data":"aee7de5cad0ccf2a4ece708d176e0e5bbd3d03443a8ec74d9e034cadecf82962"} Sep 10 22:19:38.132740 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:38.132567 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" event={"ID":"82cb99cc-1e95-43ee-8209-f9113ed0e5ea","Type":"ContainerStarted","Data":"9a92b8773b704f5d506f77cc4b4f704cd61a6c1ca68cba9743772731a6db8a31"} Sep 10 22:19:38.132740 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:38.132624 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" Sep 10 22:19:38.151809 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:38.151757 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" podStartSLOduration=1.762628002 podStartE2EDuration="2.151736635s" totalImagesPullingTime="389.108633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:19:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:19:38.150744979 +0000 UTC m=+879.720234944" watchObservedRunningTime="2026-09-10 22:19:38.151736635 +0000 UTC m=+879.721226602" Sep 10 22:19:44.145719 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:44.145685 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" Sep 10 22:19:55.232197 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:55.232105 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh"] Sep 10 22:19:55.232681 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:55.232370 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh" podUID="4298b2c8-c852-4bc9-b72c-a686041af6f2" containerName="maas-api" containerID="cri-o://e7ddf2ff5d1b6ec5f7d15dbc2a164c5cd29afef92dc8c0fb3ac893ccb50b48e0" gracePeriod=30 Sep 10 22:19:55.496362 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:55.496301 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh" Sep 10 22:19:55.664225 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:55.664186 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4298b2c8-c852-4bc9-b72c-a686041af6f2-openshift-service-ca\") pod \"4298b2c8-c852-4bc9-b72c-a686041af6f2\" (UID: \"4298b2c8-c852-4bc9-b72c-a686041af6f2\") " Sep 10 22:19:55.664225 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:55.664225 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4298b2c8-c852-4bc9-b72c-a686041af6f2-maas-api-tls\") pod \"4298b2c8-c852-4bc9-b72c-a686041af6f2\" (UID: \"4298b2c8-c852-4bc9-b72c-a686041af6f2\") " Sep 10 22:19:55.664524 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:55.664288 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4298b2c8-c852-4bc9-b72c-a686041af6f2-metrics-certs\") pod \"4298b2c8-c852-4bc9-b72c-a686041af6f2\" (UID: \"4298b2c8-c852-4bc9-b72c-a686041af6f2\") " Sep 10 22:19:55.664524 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:55.664325 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88h8m\" (UniqueName: \"kubernetes.io/projected/4298b2c8-c852-4bc9-b72c-a686041af6f2-kube-api-access-88h8m\") pod \"4298b2c8-c852-4bc9-b72c-a686041af6f2\" (UID: \"4298b2c8-c852-4bc9-b72c-a686041af6f2\") " Sep 10 22:19:55.664649 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:55.664603 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4298b2c8-c852-4bc9-b72c-a686041af6f2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4298b2c8-c852-4bc9-b72c-a686041af6f2" (UID: "4298b2c8-c852-4bc9-b72c-a686041af6f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:19:55.664736 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:55.664711 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4298b2c8-c852-4bc9-b72c-a686041af6f2-openshift-service-ca\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:19:55.667002 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:55.666970 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4298b2c8-c852-4bc9-b72c-a686041af6f2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4298b2c8-c852-4bc9-b72c-a686041af6f2" (UID: "4298b2c8-c852-4bc9-b72c-a686041af6f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:19:55.667117 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:55.666997 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4298b2c8-c852-4bc9-b72c-a686041af6f2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4298b2c8-c852-4bc9-b72c-a686041af6f2" (UID: "4298b2c8-c852-4bc9-b72c-a686041af6f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:19:55.667117 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:55.667006 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4298b2c8-c852-4bc9-b72c-a686041af6f2-kube-api-access-88h8m" (OuterVolumeSpecName: "kube-api-access-88h8m") pod "4298b2c8-c852-4bc9-b72c-a686041af6f2" (UID: "4298b2c8-c852-4bc9-b72c-a686041af6f2"). InnerVolumeSpecName "kube-api-access-88h8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:19:55.765741 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:55.765639 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4298b2c8-c852-4bc9-b72c-a686041af6f2-maas-api-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:19:55.765741 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:55.765683 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4298b2c8-c852-4bc9-b72c-a686041af6f2-metrics-certs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:19:55.765741 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:55.765696 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88h8m\" (UniqueName: \"kubernetes.io/projected/4298b2c8-c852-4bc9-b72c-a686041af6f2-kube-api-access-88h8m\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:19:56.228612 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:56.228573 2564 generic.go:356] "Generic (PLEG): container finished" podID="4298b2c8-c852-4bc9-b72c-a686041af6f2" containerID="e7ddf2ff5d1b6ec5f7d15dbc2a164c5cd29afef92dc8c0fb3ac893ccb50b48e0" exitCode=0 Sep 10 22:19:56.228777 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:56.228619 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh" event={"ID":"4298b2c8-c852-4bc9-b72c-a686041af6f2","Type":"ContainerDied","Data":"e7ddf2ff5d1b6ec5f7d15dbc2a164c5cd29afef92dc8c0fb3ac893ccb50b48e0"} Sep 10 22:19:56.228777 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:56.228624 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh" Sep 10 22:19:56.228777 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:56.228658 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh" event={"ID":"4298b2c8-c852-4bc9-b72c-a686041af6f2","Type":"ContainerDied","Data":"476393bd5ba4f40c70c3d8bb8c4ac5f43775f9a90579a5009ec219130a02791b"} Sep 10 22:19:56.228777 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:56.228678 2564 scope.go:117] "RemoveContainer" containerID="e7ddf2ff5d1b6ec5f7d15dbc2a164c5cd29afef92dc8c0fb3ac893ccb50b48e0" Sep 10 22:19:56.238943 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:56.238736 2564 scope.go:117] "RemoveContainer" containerID="e7ddf2ff5d1b6ec5f7d15dbc2a164c5cd29afef92dc8c0fb3ac893ccb50b48e0" Sep 10 22:19:56.239157 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:19:56.239038 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7ddf2ff5d1b6ec5f7d15dbc2a164c5cd29afef92dc8c0fb3ac893ccb50b48e0\": container with ID starting with e7ddf2ff5d1b6ec5f7d15dbc2a164c5cd29afef92dc8c0fb3ac893ccb50b48e0 not found: ID does not exist" containerID="e7ddf2ff5d1b6ec5f7d15dbc2a164c5cd29afef92dc8c0fb3ac893ccb50b48e0" Sep 10 22:19:56.239157 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:56.239062 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7ddf2ff5d1b6ec5f7d15dbc2a164c5cd29afef92dc8c0fb3ac893ccb50b48e0"} err="failed to get container status \"e7ddf2ff5d1b6ec5f7d15dbc2a164c5cd29afef92dc8c0fb3ac893ccb50b48e0\": rpc error: code = NotFound desc = could not find container \"e7ddf2ff5d1b6ec5f7d15dbc2a164c5cd29afef92dc8c0fb3ac893ccb50b48e0\": container with ID starting with e7ddf2ff5d1b6ec5f7d15dbc2a164c5cd29afef92dc8c0fb3ac893ccb50b48e0 not found: ID does not exist" Sep 10 22:19:56.257948 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:56.257924 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh"] Sep 10 22:19:56.262060 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:56.262042 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-938f94d8-64b4b79997-447lh"] Sep 10 22:19:56.938121 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:56.938083 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4298b2c8-c852-4bc9-b72c-a686041af6f2" path="/var/lib/kubelet/pods/4298b2c8-c852-4bc9-b72c-a686041af6f2/volumes" Sep 10 22:19:59.643385 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:59.643352 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct"] Sep 10 22:19:59.643831 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:59.643810 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4298b2c8-c852-4bc9-b72c-a686041af6f2" containerName="maas-api" Sep 10 22:19:59.643831 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:59.643833 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4298b2c8-c852-4bc9-b72c-a686041af6f2" containerName="maas-api" Sep 10 22:19:59.644045 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:59.644028 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4298b2c8-c852-4bc9-b72c-a686041af6f2" containerName="maas-api" Sep 10 22:19:59.649171 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:59.649151 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct" Sep 10 22:19:59.652623 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:59.652601 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-7178ee60-serving-cert\"" Sep 10 22:19:59.660639 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:59.660550 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct"] Sep 10 22:19:59.803557 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:59.803522 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9pfk\" (UniqueName: \"kubernetes.io/projected/137383c8-c919-4106-afea-fe43024bafa2-kube-api-access-b9pfk\") pod \"maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct\" (UID: \"137383c8-c919-4106-afea-fe43024bafa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct" Sep 10 22:19:59.803726 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:59.803569 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/137383c8-c919-4106-afea-fe43024bafa2-openshift-service-ca\") pod \"maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct\" (UID: \"137383c8-c919-4106-afea-fe43024bafa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct" Sep 10 22:19:59.803726 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:59.803647 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/137383c8-c919-4106-afea-fe43024bafa2-metrics-certs\") pod \"maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct\" (UID: \"137383c8-c919-4106-afea-fe43024bafa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct" Sep 10 22:19:59.803807 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:59.803721 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/137383c8-c919-4106-afea-fe43024bafa2-maas-api-tls\") pod \"maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct\" (UID: \"137383c8-c919-4106-afea-fe43024bafa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct" Sep 10 22:19:59.905098 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:59.905023 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/137383c8-c919-4106-afea-fe43024bafa2-maas-api-tls\") pod \"maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct\" (UID: \"137383c8-c919-4106-afea-fe43024bafa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct" Sep 10 22:19:59.905252 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:59.905104 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9pfk\" (UniqueName: \"kubernetes.io/projected/137383c8-c919-4106-afea-fe43024bafa2-kube-api-access-b9pfk\") pod \"maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct\" (UID: \"137383c8-c919-4106-afea-fe43024bafa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct" Sep 10 22:19:59.905252 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:59.905128 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/137383c8-c919-4106-afea-fe43024bafa2-openshift-service-ca\") pod \"maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct\" (UID: \"137383c8-c919-4106-afea-fe43024bafa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct" Sep 10 22:19:59.905252 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:59.905163 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/137383c8-c919-4106-afea-fe43024bafa2-metrics-certs\") pod \"maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct\" (UID: \"137383c8-c919-4106-afea-fe43024bafa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct" Sep 10 22:19:59.905810 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:59.905785 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/137383c8-c919-4106-afea-fe43024bafa2-openshift-service-ca\") pod \"maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct\" (UID: \"137383c8-c919-4106-afea-fe43024bafa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct" Sep 10 22:19:59.907468 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:59.907442 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/137383c8-c919-4106-afea-fe43024bafa2-metrics-certs\") pod \"maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct\" (UID: \"137383c8-c919-4106-afea-fe43024bafa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct" Sep 10 22:19:59.907561 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:59.907507 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/137383c8-c919-4106-afea-fe43024bafa2-maas-api-tls\") pod \"maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct\" (UID: \"137383c8-c919-4106-afea-fe43024bafa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct" Sep 10 22:19:59.913121 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:59.913101 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9pfk\" (UniqueName: \"kubernetes.io/projected/137383c8-c919-4106-afea-fe43024bafa2-kube-api-access-b9pfk\") pod \"maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct\" (UID: \"137383c8-c919-4106-afea-fe43024bafa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct" Sep 10 22:19:59.961116 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:19:59.961094 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct" Sep 10 22:20:00.240528 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:00.240439 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct"] Sep 10 22:20:00.244241 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:20:00.244210 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137383c8_c919_4106_afea_fe43024bafa2.slice/crio-3a53d7383065e4eb932933095cfcac6b6ddf6960ea77e7fc54c44bfa8595f44c WatchSource:0}: Error finding container 3a53d7383065e4eb932933095cfcac6b6ddf6960ea77e7fc54c44bfa8595f44c: Status 404 returned error can't find the container with id 3a53d7383065e4eb932933095cfcac6b6ddf6960ea77e7fc54c44bfa8595f44c Sep 10 22:20:01.264139 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:01.264106 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct" event={"ID":"137383c8-c919-4106-afea-fe43024bafa2","Type":"ContainerStarted","Data":"6df47a19c28cf8848d0b0814fc476f4107401540d2aefa436151d201858308f9"} Sep 10 22:20:01.264543 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:01.264162 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct" event={"ID":"137383c8-c919-4106-afea-fe43024bafa2","Type":"ContainerStarted","Data":"3a53d7383065e4eb932933095cfcac6b6ddf6960ea77e7fc54c44bfa8595f44c"} Sep 10 22:20:01.264543 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:01.264271 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct" Sep 10 22:20:01.281238 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:01.281186 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct" podStartSLOduration=1.938915493 podStartE2EDuration="2.281169162s" totalImagesPullingTime="342.253669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:20:01.280630731 +0000 UTC m=+902.850120694" watchObservedRunningTime="2026-09-10 22:20:01.281169162 +0000 UTC m=+902.850659129" Sep 10 22:20:07.275489 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:07.275462 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct" Sep 10 22:20:30.954079 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:30.954033 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct"] Sep 10 22:20:30.954686 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:30.954291 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct" podUID="137383c8-c919-4106-afea-fe43024bafa2" containerName="maas-api" containerID="cri-o://6df47a19c28cf8848d0b0814fc476f4107401540d2aefa436151d201858308f9" gracePeriod=30 Sep 10 22:20:31.208063 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:31.207999 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct" Sep 10 22:20:31.309139 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:31.309103 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9pfk\" (UniqueName: \"kubernetes.io/projected/137383c8-c919-4106-afea-fe43024bafa2-kube-api-access-b9pfk\") pod \"137383c8-c919-4106-afea-fe43024bafa2\" (UID: \"137383c8-c919-4106-afea-fe43024bafa2\") " Sep 10 22:20:31.309343 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:31.309151 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/137383c8-c919-4106-afea-fe43024bafa2-openshift-service-ca\") pod \"137383c8-c919-4106-afea-fe43024bafa2\" (UID: \"137383c8-c919-4106-afea-fe43024bafa2\") " Sep 10 22:20:31.309343 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:31.309235 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/137383c8-c919-4106-afea-fe43024bafa2-metrics-certs\") pod \"137383c8-c919-4106-afea-fe43024bafa2\" (UID: \"137383c8-c919-4106-afea-fe43024bafa2\") " Sep 10 22:20:31.309343 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:31.309260 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/137383c8-c919-4106-afea-fe43024bafa2-maas-api-tls\") pod \"137383c8-c919-4106-afea-fe43024bafa2\" (UID: \"137383c8-c919-4106-afea-fe43024bafa2\") " Sep 10 22:20:31.309686 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:31.309653 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137383c8-c919-4106-afea-fe43024bafa2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "137383c8-c919-4106-afea-fe43024bafa2" (UID: "137383c8-c919-4106-afea-fe43024bafa2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:20:31.313166 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:31.313134 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137383c8-c919-4106-afea-fe43024bafa2-kube-api-access-b9pfk" (OuterVolumeSpecName: "kube-api-access-b9pfk") pod "137383c8-c919-4106-afea-fe43024bafa2" (UID: "137383c8-c919-4106-afea-fe43024bafa2"). InnerVolumeSpecName "kube-api-access-b9pfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:20:31.314451 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:31.314422 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137383c8-c919-4106-afea-fe43024bafa2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "137383c8-c919-4106-afea-fe43024bafa2" (UID: "137383c8-c919-4106-afea-fe43024bafa2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:20:31.314451 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:31.314440 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137383c8-c919-4106-afea-fe43024bafa2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "137383c8-c919-4106-afea-fe43024bafa2" (UID: "137383c8-c919-4106-afea-fe43024bafa2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:20:31.410493 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:31.410461 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/137383c8-c919-4106-afea-fe43024bafa2-metrics-certs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:20:31.410493 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:31.410487 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/137383c8-c919-4106-afea-fe43024bafa2-maas-api-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:20:31.410493 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:31.410496 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9pfk\" (UniqueName: \"kubernetes.io/projected/137383c8-c919-4106-afea-fe43024bafa2-kube-api-access-b9pfk\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:20:31.410740 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:31.410505 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/137383c8-c919-4106-afea-fe43024bafa2-openshift-service-ca\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:20:31.418822 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:31.418791 2564 generic.go:356] "Generic (PLEG): container finished" podID="137383c8-c919-4106-afea-fe43024bafa2" containerID="6df47a19c28cf8848d0b0814fc476f4107401540d2aefa436151d201858308f9" exitCode=0 Sep 10 22:20:31.419017 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:31.418834 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct" Sep 10 22:20:31.419017 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:31.418874 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct" event={"ID":"137383c8-c919-4106-afea-fe43024bafa2","Type":"ContainerDied","Data":"6df47a19c28cf8848d0b0814fc476f4107401540d2aefa436151d201858308f9"} Sep 10 22:20:31.419017 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:31.418938 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct" event={"ID":"137383c8-c919-4106-afea-fe43024bafa2","Type":"ContainerDied","Data":"3a53d7383065e4eb932933095cfcac6b6ddf6960ea77e7fc54c44bfa8595f44c"} Sep 10 22:20:31.419017 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:31.418964 2564 scope.go:117] "RemoveContainer" containerID="6df47a19c28cf8848d0b0814fc476f4107401540d2aefa436151d201858308f9" Sep 10 22:20:31.428626 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:31.428611 2564 scope.go:117] "RemoveContainer" containerID="6df47a19c28cf8848d0b0814fc476f4107401540d2aefa436151d201858308f9" Sep 10 22:20:31.428933 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:20:31.428911 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df47a19c28cf8848d0b0814fc476f4107401540d2aefa436151d201858308f9\": container with ID starting with 6df47a19c28cf8848d0b0814fc476f4107401540d2aefa436151d201858308f9 not found: ID does not exist" containerID="6df47a19c28cf8848d0b0814fc476f4107401540d2aefa436151d201858308f9" Sep 10 22:20:31.428993 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:31.428943 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df47a19c28cf8848d0b0814fc476f4107401540d2aefa436151d201858308f9"} err="failed to get container status \"6df47a19c28cf8848d0b0814fc476f4107401540d2aefa436151d201858308f9\": rpc error: code = NotFound desc = could not find container \"6df47a19c28cf8848d0b0814fc476f4107401540d2aefa436151d201858308f9\": container with ID starting with 6df47a19c28cf8848d0b0814fc476f4107401540d2aefa436151d201858308f9 not found: ID does not exist" Sep 10 22:20:31.440595 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:31.440573 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct"] Sep 10 22:20:31.445862 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:31.445840 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7178ee60-c4d87d84-9tbct"] Sep 10 22:20:32.937491 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:32.937457 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="137383c8-c919-4106-afea-fe43024bafa2" path="/var/lib/kubelet/pods/137383c8-c919-4106-afea-fe43024bafa2/volumes" Sep 10 22:20:35.694599 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:35.694567 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc"] Sep 10 22:20:35.695219 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:35.695198 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="137383c8-c919-4106-afea-fe43024bafa2" containerName="maas-api" Sep 10 22:20:35.695309 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:35.695222 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="137383c8-c919-4106-afea-fe43024bafa2" containerName="maas-api" Sep 10 22:20:35.695362 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:35.695317 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="137383c8-c919-4106-afea-fe43024bafa2" containerName="maas-api" Sep 10 22:20:35.700241 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:35.700219 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc" Sep 10 22:20:35.702784 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:35.702761 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-178c50fc-serving-cert\"" Sep 10 22:20:35.708084 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:35.707749 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc"] Sep 10 22:20:35.854524 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:35.854490 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bgm5\" (UniqueName: \"kubernetes.io/projected/dcbc8281-9df9-4b8a-8e2c-bd37e801f128-kube-api-access-6bgm5\") pod \"maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc\" (UID: \"dcbc8281-9df9-4b8a-8e2c-bd37e801f128\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc" Sep 10 22:20:35.854723 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:35.854553 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcbc8281-9df9-4b8a-8e2c-bd37e801f128-maas-api-tls\") pod \"maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc\" (UID: \"dcbc8281-9df9-4b8a-8e2c-bd37e801f128\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc" Sep 10 22:20:35.854723 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:35.854576 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbc8281-9df9-4b8a-8e2c-bd37e801f128-openshift-service-ca\") pod \"maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc\" (UID: \"dcbc8281-9df9-4b8a-8e2c-bd37e801f128\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc" Sep 10 22:20:35.854723 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:35.854611 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcbc8281-9df9-4b8a-8e2c-bd37e801f128-metrics-certs\") pod \"maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc\" (UID: \"dcbc8281-9df9-4b8a-8e2c-bd37e801f128\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc" Sep 10 22:20:35.955747 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:35.955663 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcbc8281-9df9-4b8a-8e2c-bd37e801f128-metrics-certs\") pod \"maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc\" (UID: \"dcbc8281-9df9-4b8a-8e2c-bd37e801f128\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc" Sep 10 22:20:35.955934 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:35.955780 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bgm5\" (UniqueName: \"kubernetes.io/projected/dcbc8281-9df9-4b8a-8e2c-bd37e801f128-kube-api-access-6bgm5\") pod \"maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc\" (UID: \"dcbc8281-9df9-4b8a-8e2c-bd37e801f128\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc" Sep 10 22:20:35.955934 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:35.955844 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcbc8281-9df9-4b8a-8e2c-bd37e801f128-maas-api-tls\") pod \"maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc\" (UID: \"dcbc8281-9df9-4b8a-8e2c-bd37e801f128\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc" Sep 10 22:20:35.955934 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:35.955869 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbc8281-9df9-4b8a-8e2c-bd37e801f128-openshift-service-ca\") pod \"maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc\" (UID: \"dcbc8281-9df9-4b8a-8e2c-bd37e801f128\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc" Sep 10 22:20:35.956611 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:35.956585 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbc8281-9df9-4b8a-8e2c-bd37e801f128-openshift-service-ca\") pod \"maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc\" (UID: \"dcbc8281-9df9-4b8a-8e2c-bd37e801f128\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc" Sep 10 22:20:35.958311 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:35.958268 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcbc8281-9df9-4b8a-8e2c-bd37e801f128-maas-api-tls\") pod \"maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc\" (UID: \"dcbc8281-9df9-4b8a-8e2c-bd37e801f128\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc" Sep 10 22:20:35.958399 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:35.958316 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcbc8281-9df9-4b8a-8e2c-bd37e801f128-metrics-certs\") pod \"maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc\" (UID: \"dcbc8281-9df9-4b8a-8e2c-bd37e801f128\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc" Sep 10 22:20:35.963393 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:35.963374 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bgm5\" (UniqueName: \"kubernetes.io/projected/dcbc8281-9df9-4b8a-8e2c-bd37e801f128-kube-api-access-6bgm5\") pod \"maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc\" (UID: \"dcbc8281-9df9-4b8a-8e2c-bd37e801f128\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc" Sep 10 22:20:36.013242 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:36.013195 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc" Sep 10 22:20:36.287644 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:36.287616 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc"] Sep 10 22:20:36.289456 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:20:36.289420 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbc8281_9df9_4b8a_8e2c_bd37e801f128.slice/crio-d5c922117af72f36b90e2159f3ea9442d1e93815a7226fd2a8a0a403ed5a0ed2 WatchSource:0}: Error finding container d5c922117af72f36b90e2159f3ea9442d1e93815a7226fd2a8a0a403ed5a0ed2: Status 404 returned error can't find the container with id d5c922117af72f36b90e2159f3ea9442d1e93815a7226fd2a8a0a403ed5a0ed2 Sep 10 22:20:36.447340 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:36.447289 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc" event={"ID":"dcbc8281-9df9-4b8a-8e2c-bd37e801f128","Type":"ContainerStarted","Data":"d5c922117af72f36b90e2159f3ea9442d1e93815a7226fd2a8a0a403ed5a0ed2"} Sep 10 22:20:37.454879 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:37.454838 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc" event={"ID":"dcbc8281-9df9-4b8a-8e2c-bd37e801f128","Type":"ContainerStarted","Data":"373660f04c173b1702706ea31c8a2aaa58674144ab832d95f7bc6c50f2ac78d3"} Sep 10 22:20:37.455362 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:37.454922 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc" Sep 10 22:20:37.470329 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:37.470281 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc" podStartSLOduration=2.112185535 podStartE2EDuration="2.470265808s" totalImagesPullingTime="358.080273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:20:37.469786488 +0000 UTC m=+939.039276455" watchObservedRunningTime="2026-09-10 22:20:37.470265808 +0000 UTC m=+939.039755771" Sep 10 22:20:43.466503 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:20:43.466475 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc" Sep 10 22:21:06.207398 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:06.207360 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc"] Sep 10 22:21:06.207962 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:06.207613 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc" podUID="dcbc8281-9df9-4b8a-8e2c-bd37e801f128" containerName="maas-api" containerID="cri-o://373660f04c173b1702706ea31c8a2aaa58674144ab832d95f7bc6c50f2ac78d3" gracePeriod=30 Sep 10 22:21:06.457382 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:06.457353 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc" Sep 10 22:21:06.538990 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:06.538920 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbc8281-9df9-4b8a-8e2c-bd37e801f128-openshift-service-ca\") pod \"dcbc8281-9df9-4b8a-8e2c-bd37e801f128\" (UID: \"dcbc8281-9df9-4b8a-8e2c-bd37e801f128\") " Sep 10 22:21:06.538990 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:06.538968 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bgm5\" (UniqueName: \"kubernetes.io/projected/dcbc8281-9df9-4b8a-8e2c-bd37e801f128-kube-api-access-6bgm5\") pod \"dcbc8281-9df9-4b8a-8e2c-bd37e801f128\" (UID: \"dcbc8281-9df9-4b8a-8e2c-bd37e801f128\") " Sep 10 22:21:06.539181 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:06.539018 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcbc8281-9df9-4b8a-8e2c-bd37e801f128-metrics-certs\") pod \"dcbc8281-9df9-4b8a-8e2c-bd37e801f128\" (UID: \"dcbc8281-9df9-4b8a-8e2c-bd37e801f128\") " Sep 10 22:21:06.539181 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:06.539087 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcbc8281-9df9-4b8a-8e2c-bd37e801f128-maas-api-tls\") pod \"dcbc8281-9df9-4b8a-8e2c-bd37e801f128\" (UID: \"dcbc8281-9df9-4b8a-8e2c-bd37e801f128\") " Sep 10 22:21:06.539332 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:06.539308 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbc8281-9df9-4b8a-8e2c-bd37e801f128-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dcbc8281-9df9-4b8a-8e2c-bd37e801f128" (UID: "dcbc8281-9df9-4b8a-8e2c-bd37e801f128"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:21:06.539427 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:06.539412 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbc8281-9df9-4b8a-8e2c-bd37e801f128-openshift-service-ca\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:21:06.541198 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:06.541168 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcbc8281-9df9-4b8a-8e2c-bd37e801f128-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dcbc8281-9df9-4b8a-8e2c-bd37e801f128" (UID: "dcbc8281-9df9-4b8a-8e2c-bd37e801f128"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:21:06.541198 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:06.541174 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbc8281-9df9-4b8a-8e2c-bd37e801f128-kube-api-access-6bgm5" (OuterVolumeSpecName: "kube-api-access-6bgm5") pod "dcbc8281-9df9-4b8a-8e2c-bd37e801f128" (UID: "dcbc8281-9df9-4b8a-8e2c-bd37e801f128"). InnerVolumeSpecName "kube-api-access-6bgm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:21:06.541342 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:06.541271 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcbc8281-9df9-4b8a-8e2c-bd37e801f128-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dcbc8281-9df9-4b8a-8e2c-bd37e801f128" (UID: "dcbc8281-9df9-4b8a-8e2c-bd37e801f128"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:21:06.601093 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:06.601056 2564 generic.go:356] "Generic (PLEG): container finished" podID="dcbc8281-9df9-4b8a-8e2c-bd37e801f128" containerID="373660f04c173b1702706ea31c8a2aaa58674144ab832d95f7bc6c50f2ac78d3" exitCode=0 Sep 10 22:21:06.601250 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:06.601100 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc" Sep 10 22:21:06.601250 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:06.601118 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc" event={"ID":"dcbc8281-9df9-4b8a-8e2c-bd37e801f128","Type":"ContainerDied","Data":"373660f04c173b1702706ea31c8a2aaa58674144ab832d95f7bc6c50f2ac78d3"} Sep 10 22:21:06.601250 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:06.601169 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc" event={"ID":"dcbc8281-9df9-4b8a-8e2c-bd37e801f128","Type":"ContainerDied","Data":"d5c922117af72f36b90e2159f3ea9442d1e93815a7226fd2a8a0a403ed5a0ed2"} Sep 10 22:21:06.601250 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:06.601199 2564 scope.go:117] "RemoveContainer" containerID="373660f04c173b1702706ea31c8a2aaa58674144ab832d95f7bc6c50f2ac78d3" Sep 10 22:21:06.610392 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:06.610378 2564 scope.go:117] "RemoveContainer" containerID="373660f04c173b1702706ea31c8a2aaa58674144ab832d95f7bc6c50f2ac78d3" Sep 10 22:21:06.610636 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:21:06.610617 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"373660f04c173b1702706ea31c8a2aaa58674144ab832d95f7bc6c50f2ac78d3\": container with ID starting with 373660f04c173b1702706ea31c8a2aaa58674144ab832d95f7bc6c50f2ac78d3 not found: ID does not exist" containerID="373660f04c173b1702706ea31c8a2aaa58674144ab832d95f7bc6c50f2ac78d3" Sep 10 22:21:06.610673 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:06.610645 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"373660f04c173b1702706ea31c8a2aaa58674144ab832d95f7bc6c50f2ac78d3"} err="failed to get container status \"373660f04c173b1702706ea31c8a2aaa58674144ab832d95f7bc6c50f2ac78d3\": rpc error: code = NotFound desc = could not find container \"373660f04c173b1702706ea31c8a2aaa58674144ab832d95f7bc6c50f2ac78d3\": container with ID starting with 373660f04c173b1702706ea31c8a2aaa58674144ab832d95f7bc6c50f2ac78d3 not found: ID does not exist" Sep 10 22:21:06.621513 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:06.621493 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc"] Sep 10 22:21:06.627542 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:06.627522 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-178c50fc-bf66f68c7-kmsmc"] Sep 10 22:21:06.640873 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:06.640854 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bgm5\" (UniqueName: \"kubernetes.io/projected/dcbc8281-9df9-4b8a-8e2c-bd37e801f128-kube-api-access-6bgm5\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:21:06.640958 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:06.640875 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcbc8281-9df9-4b8a-8e2c-bd37e801f128-metrics-certs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:21:06.640958 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:06.640885 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcbc8281-9df9-4b8a-8e2c-bd37e801f128-maas-api-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:21:06.938146 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:06.938111 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcbc8281-9df9-4b8a-8e2c-bd37e801f128" path="/var/lib/kubelet/pods/dcbc8281-9df9-4b8a-8e2c-bd37e801f128/volumes" Sep 10 22:21:12.005949 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:12.005913 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg"] Sep 10 22:21:12.006591 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:12.006565 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcbc8281-9df9-4b8a-8e2c-bd37e801f128" containerName="maas-api" Sep 10 22:21:12.006700 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:12.006593 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcbc8281-9df9-4b8a-8e2c-bd37e801f128" containerName="maas-api" Sep 10 22:21:12.006757 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:12.006712 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcbc8281-9df9-4b8a-8e2c-bd37e801f128" containerName="maas-api" Sep 10 22:21:12.011640 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:12.011618 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg" Sep 10 22:21:12.014233 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:12.014206 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-2057d50e-serving-cert\"" Sep 10 22:21:12.022147 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:12.021577 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg"] Sep 10 22:21:12.097354 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:12.097305 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn5gk\" (UniqueName: \"kubernetes.io/projected/dfc465df-6993-45e6-ae57-7262c4d0ab8b-kube-api-access-zn5gk\") pod \"maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg\" (UID: \"dfc465df-6993-45e6-ae57-7262c4d0ab8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg" Sep 10 22:21:12.097519 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:12.097501 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc465df-6993-45e6-ae57-7262c4d0ab8b-openshift-service-ca\") pod \"maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg\" (UID: \"dfc465df-6993-45e6-ae57-7262c4d0ab8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg" Sep 10 22:21:12.097598 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:12.097530 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc465df-6993-45e6-ae57-7262c4d0ab8b-maas-api-tls\") pod \"maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg\" (UID: \"dfc465df-6993-45e6-ae57-7262c4d0ab8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg" Sep 10 22:21:12.098205 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:12.098140 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc465df-6993-45e6-ae57-7262c4d0ab8b-metrics-certs\") pod \"maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg\" (UID: \"dfc465df-6993-45e6-ae57-7262c4d0ab8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg" Sep 10 22:21:12.198980 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:12.198943 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn5gk\" (UniqueName: \"kubernetes.io/projected/dfc465df-6993-45e6-ae57-7262c4d0ab8b-kube-api-access-zn5gk\") pod \"maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg\" (UID: \"dfc465df-6993-45e6-ae57-7262c4d0ab8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg" Sep 10 22:21:12.199155 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:12.199044 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc465df-6993-45e6-ae57-7262c4d0ab8b-openshift-service-ca\") pod \"maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg\" (UID: \"dfc465df-6993-45e6-ae57-7262c4d0ab8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg" Sep 10 22:21:12.199155 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:12.199064 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc465df-6993-45e6-ae57-7262c4d0ab8b-maas-api-tls\") pod \"maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg\" (UID: \"dfc465df-6993-45e6-ae57-7262c4d0ab8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg" Sep 10 22:21:12.199155 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:12.199090 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc465df-6993-45e6-ae57-7262c4d0ab8b-metrics-certs\") pod \"maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg\" (UID: \"dfc465df-6993-45e6-ae57-7262c4d0ab8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg" Sep 10 22:21:12.199723 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:12.199684 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc465df-6993-45e6-ae57-7262c4d0ab8b-openshift-service-ca\") pod \"maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg\" (UID: \"dfc465df-6993-45e6-ae57-7262c4d0ab8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg" Sep 10 22:21:12.201573 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:12.201540 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc465df-6993-45e6-ae57-7262c4d0ab8b-metrics-certs\") pod \"maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg\" (UID: \"dfc465df-6993-45e6-ae57-7262c4d0ab8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg" Sep 10 22:21:12.201683 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:12.201604 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc465df-6993-45e6-ae57-7262c4d0ab8b-maas-api-tls\") pod \"maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg\" (UID: \"dfc465df-6993-45e6-ae57-7262c4d0ab8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg" Sep 10 22:21:12.208336 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:12.208316 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn5gk\" (UniqueName: \"kubernetes.io/projected/dfc465df-6993-45e6-ae57-7262c4d0ab8b-kube-api-access-zn5gk\") pod \"maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg\" (UID: \"dfc465df-6993-45e6-ae57-7262c4d0ab8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg" Sep 10 22:21:12.336055 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:12.335971 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg" Sep 10 22:21:12.618404 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:12.618381 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg"] Sep 10 22:21:12.620496 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:21:12.620467 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc465df_6993_45e6_ae57_7262c4d0ab8b.slice/crio-8e98bbab304d380a247ed2bd0585c2c84ca2224cf2b3718e6f11c31ada624581 WatchSource:0}: Error finding container 8e98bbab304d380a247ed2bd0585c2c84ca2224cf2b3718e6f11c31ada624581: Status 404 returned error can't find the container with id 8e98bbab304d380a247ed2bd0585c2c84ca2224cf2b3718e6f11c31ada624581 Sep 10 22:21:12.636176 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:12.636153 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg" event={"ID":"dfc465df-6993-45e6-ae57-7262c4d0ab8b","Type":"ContainerStarted","Data":"8e98bbab304d380a247ed2bd0585c2c84ca2224cf2b3718e6f11c31ada624581"} Sep 10 22:21:13.642857 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:13.642820 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg" event={"ID":"dfc465df-6993-45e6-ae57-7262c4d0ab8b","Type":"ContainerStarted","Data":"0ccd0b3f174807954f71a2c0ec4e705ba5a438e178c41f9d7fb67a15a620f86e"} Sep 10 22:21:13.643305 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:13.642916 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg" Sep 10 22:21:13.662431 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:13.662375 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg" podStartSLOduration=2.208173812 podStartE2EDuration="2.662360344s" totalImagesPullingTime="454.186532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:21:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:21:13.660373018 +0000 UTC m=+975.229862981" watchObservedRunningTime="2026-09-10 22:21:13.662360344 +0000 UTC m=+975.231850307" Sep 10 22:21:19.655855 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:19.655827 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg" Sep 10 22:21:38.922148 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:38.922112 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb"] Sep 10 22:21:38.922716 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:38.922369 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" podUID="d11d98c9-655d-46f4-b5ce-51a850d64a25" containerName="maas-api" containerID="cri-o://c0669f1559eab505fc649728fcdeece92f6abb07120dddb278c05fdbc3de65fd" gracePeriod=30 Sep 10 22:21:39.168228 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:39.168203 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" Sep 10 22:21:39.253129 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:39.253049 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzs8d\" (UniqueName: \"kubernetes.io/projected/d11d98c9-655d-46f4-b5ce-51a850d64a25-kube-api-access-jzs8d\") pod \"d11d98c9-655d-46f4-b5ce-51a850d64a25\" (UID: \"d11d98c9-655d-46f4-b5ce-51a850d64a25\") " Sep 10 22:21:39.253129 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:39.253124 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d11d98c9-655d-46f4-b5ce-51a850d64a25-openshift-service-ca\") pod \"d11d98c9-655d-46f4-b5ce-51a850d64a25\" (UID: \"d11d98c9-655d-46f4-b5ce-51a850d64a25\") " Sep 10 22:21:39.253362 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:39.253153 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d11d98c9-655d-46f4-b5ce-51a850d64a25-maas-api-tls\") pod \"d11d98c9-655d-46f4-b5ce-51a850d64a25\" (UID: \"d11d98c9-655d-46f4-b5ce-51a850d64a25\") " Sep 10 22:21:39.253362 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:39.253173 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d11d98c9-655d-46f4-b5ce-51a850d64a25-metrics-certs\") pod \"d11d98c9-655d-46f4-b5ce-51a850d64a25\" (UID: \"d11d98c9-655d-46f4-b5ce-51a850d64a25\") " Sep 10 22:21:39.253578 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:39.253543 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d11d98c9-655d-46f4-b5ce-51a850d64a25-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d11d98c9-655d-46f4-b5ce-51a850d64a25" (UID: "d11d98c9-655d-46f4-b5ce-51a850d64a25"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:21:39.255373 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:39.255344 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d11d98c9-655d-46f4-b5ce-51a850d64a25-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d11d98c9-655d-46f4-b5ce-51a850d64a25" (UID: "d11d98c9-655d-46f4-b5ce-51a850d64a25"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:21:39.255373 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:39.255362 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d11d98c9-655d-46f4-b5ce-51a850d64a25-kube-api-access-jzs8d" (OuterVolumeSpecName: "kube-api-access-jzs8d") pod "d11d98c9-655d-46f4-b5ce-51a850d64a25" (UID: "d11d98c9-655d-46f4-b5ce-51a850d64a25"). InnerVolumeSpecName "kube-api-access-jzs8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:21:39.255505 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:39.255375 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d11d98c9-655d-46f4-b5ce-51a850d64a25-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d11d98c9-655d-46f4-b5ce-51a850d64a25" (UID: "d11d98c9-655d-46f4-b5ce-51a850d64a25"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:21:39.354233 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:39.354191 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzs8d\" (UniqueName: \"kubernetes.io/projected/d11d98c9-655d-46f4-b5ce-51a850d64a25-kube-api-access-jzs8d\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:21:39.354233 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:39.354228 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d11d98c9-655d-46f4-b5ce-51a850d64a25-openshift-service-ca\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:21:39.354233 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:39.354240 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d11d98c9-655d-46f4-b5ce-51a850d64a25-maas-api-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:21:39.354466 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:39.354252 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d11d98c9-655d-46f4-b5ce-51a850d64a25-metrics-certs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:21:39.782532 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:39.782498 2564 generic.go:356] "Generic (PLEG): container finished" podID="d11d98c9-655d-46f4-b5ce-51a850d64a25" containerID="c0669f1559eab505fc649728fcdeece92f6abb07120dddb278c05fdbc3de65fd" exitCode=0 Sep 10 22:21:39.782711 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:39.782548 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" Sep 10 22:21:39.782711 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:39.782593 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" event={"ID":"d11d98c9-655d-46f4-b5ce-51a850d64a25","Type":"ContainerDied","Data":"c0669f1559eab505fc649728fcdeece92f6abb07120dddb278c05fdbc3de65fd"} Sep 10 22:21:39.782711 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:39.782651 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb" event={"ID":"d11d98c9-655d-46f4-b5ce-51a850d64a25","Type":"ContainerDied","Data":"cd0cb27ae4df3736d6abec70bc85fbd32cadfa30bdcf9e7d9925c2d107cb9498"} Sep 10 22:21:39.782711 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:39.782683 2564 scope.go:117] "RemoveContainer" containerID="c0669f1559eab505fc649728fcdeece92f6abb07120dddb278c05fdbc3de65fd" Sep 10 22:21:39.795423 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:39.795391 2564 scope.go:117] "RemoveContainer" containerID="c0669f1559eab505fc649728fcdeece92f6abb07120dddb278c05fdbc3de65fd" Sep 10 22:21:39.795925 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:21:39.795880 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0669f1559eab505fc649728fcdeece92f6abb07120dddb278c05fdbc3de65fd\": container with ID starting with c0669f1559eab505fc649728fcdeece92f6abb07120dddb278c05fdbc3de65fd not found: ID does not exist" containerID="c0669f1559eab505fc649728fcdeece92f6abb07120dddb278c05fdbc3de65fd" Sep 10 22:21:39.796041 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:39.795928 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0669f1559eab505fc649728fcdeece92f6abb07120dddb278c05fdbc3de65fd"} err="failed to get container status \"c0669f1559eab505fc649728fcdeece92f6abb07120dddb278c05fdbc3de65fd\": rpc error: code = NotFound desc = could not find container \"c0669f1559eab505fc649728fcdeece92f6abb07120dddb278c05fdbc3de65fd\": container with ID starting with c0669f1559eab505fc649728fcdeece92f6abb07120dddb278c05fdbc3de65fd not found: ID does not exist" Sep 10 22:21:39.806053 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:39.806029 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb"] Sep 10 22:21:39.809986 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:39.809966 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-44dc54-6b45894bc7-c74pb"] Sep 10 22:21:40.937804 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:40.937771 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d11d98c9-655d-46f4-b5ce-51a850d64a25" path="/var/lib/kubelet/pods/d11d98c9-655d-46f4-b5ce-51a850d64a25/volumes" Sep 10 22:21:42.808033 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:42.807999 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg"] Sep 10 22:21:42.808531 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:42.808300 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg" podUID="dfc465df-6993-45e6-ae57-7262c4d0ab8b" containerName="maas-api" containerID="cri-o://0ccd0b3f174807954f71a2c0ec4e705ba5a438e178c41f9d7fb67a15a620f86e" gracePeriod=30 Sep 10 22:21:43.064476 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:43.064408 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg" Sep 10 22:21:43.191502 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:43.191471 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc465df-6993-45e6-ae57-7262c4d0ab8b-maas-api-tls\") pod \"dfc465df-6993-45e6-ae57-7262c4d0ab8b\" (UID: \"dfc465df-6993-45e6-ae57-7262c4d0ab8b\") " Sep 10 22:21:43.191687 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:43.191544 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc465df-6993-45e6-ae57-7262c4d0ab8b-metrics-certs\") pod \"dfc465df-6993-45e6-ae57-7262c4d0ab8b\" (UID: \"dfc465df-6993-45e6-ae57-7262c4d0ab8b\") " Sep 10 22:21:43.191687 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:43.191572 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn5gk\" (UniqueName: \"kubernetes.io/projected/dfc465df-6993-45e6-ae57-7262c4d0ab8b-kube-api-access-zn5gk\") pod \"dfc465df-6993-45e6-ae57-7262c4d0ab8b\" (UID: \"dfc465df-6993-45e6-ae57-7262c4d0ab8b\") " Sep 10 22:21:43.191687 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:43.191594 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc465df-6993-45e6-ae57-7262c4d0ab8b-openshift-service-ca\") pod \"dfc465df-6993-45e6-ae57-7262c4d0ab8b\" (UID: \"dfc465df-6993-45e6-ae57-7262c4d0ab8b\") " Sep 10 22:21:43.192077 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:43.192040 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc465df-6993-45e6-ae57-7262c4d0ab8b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dfc465df-6993-45e6-ae57-7262c4d0ab8b" (UID: "dfc465df-6993-45e6-ae57-7262c4d0ab8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:21:43.193640 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:43.193606 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfc465df-6993-45e6-ae57-7262c4d0ab8b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dfc465df-6993-45e6-ae57-7262c4d0ab8b" (UID: "dfc465df-6993-45e6-ae57-7262c4d0ab8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:21:43.194089 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:43.194061 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfc465df-6993-45e6-ae57-7262c4d0ab8b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dfc465df-6993-45e6-ae57-7262c4d0ab8b" (UID: "dfc465df-6993-45e6-ae57-7262c4d0ab8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:21:43.194089 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:43.194072 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc465df-6993-45e6-ae57-7262c4d0ab8b-kube-api-access-zn5gk" (OuterVolumeSpecName: "kube-api-access-zn5gk") pod "dfc465df-6993-45e6-ae57-7262c4d0ab8b" (UID: "dfc465df-6993-45e6-ae57-7262c4d0ab8b"). InnerVolumeSpecName "kube-api-access-zn5gk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:21:43.292359 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:43.292331 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc465df-6993-45e6-ae57-7262c4d0ab8b-metrics-certs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:21:43.292359 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:43.292356 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn5gk\" (UniqueName: \"kubernetes.io/projected/dfc465df-6993-45e6-ae57-7262c4d0ab8b-kube-api-access-zn5gk\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:21:43.292359 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:43.292366 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc465df-6993-45e6-ae57-7262c4d0ab8b-openshift-service-ca\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:21:43.292585 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:43.292374 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc465df-6993-45e6-ae57-7262c4d0ab8b-maas-api-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:21:43.806823 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:43.806784 2564 generic.go:356] "Generic (PLEG): container finished" podID="dfc465df-6993-45e6-ae57-7262c4d0ab8b" containerID="0ccd0b3f174807954f71a2c0ec4e705ba5a438e178c41f9d7fb67a15a620f86e" exitCode=0 Sep 10 22:21:43.807021 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:43.806831 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg" Sep 10 22:21:43.807021 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:43.806837 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg" event={"ID":"dfc465df-6993-45e6-ae57-7262c4d0ab8b","Type":"ContainerDied","Data":"0ccd0b3f174807954f71a2c0ec4e705ba5a438e178c41f9d7fb67a15a620f86e"} Sep 10 22:21:43.807021 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:43.806884 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg" event={"ID":"dfc465df-6993-45e6-ae57-7262c4d0ab8b","Type":"ContainerDied","Data":"8e98bbab304d380a247ed2bd0585c2c84ca2224cf2b3718e6f11c31ada624581"} Sep 10 22:21:43.807021 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:43.806938 2564 scope.go:117] "RemoveContainer" containerID="0ccd0b3f174807954f71a2c0ec4e705ba5a438e178c41f9d7fb67a15a620f86e" Sep 10 22:21:43.816924 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:43.816805 2564 scope.go:117] "RemoveContainer" containerID="0ccd0b3f174807954f71a2c0ec4e705ba5a438e178c41f9d7fb67a15a620f86e" Sep 10 22:21:43.817234 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:21:43.817179 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ccd0b3f174807954f71a2c0ec4e705ba5a438e178c41f9d7fb67a15a620f86e\": container with ID starting with 0ccd0b3f174807954f71a2c0ec4e705ba5a438e178c41f9d7fb67a15a620f86e not found: ID does not exist" containerID="0ccd0b3f174807954f71a2c0ec4e705ba5a438e178c41f9d7fb67a15a620f86e" Sep 10 22:21:43.817234 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:43.817217 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ccd0b3f174807954f71a2c0ec4e705ba5a438e178c41f9d7fb67a15a620f86e"} err="failed to get container status \"0ccd0b3f174807954f71a2c0ec4e705ba5a438e178c41f9d7fb67a15a620f86e\": rpc error: code = NotFound desc = could not find container \"0ccd0b3f174807954f71a2c0ec4e705ba5a438e178c41f9d7fb67a15a620f86e\": container with ID starting with 0ccd0b3f174807954f71a2c0ec4e705ba5a438e178c41f9d7fb67a15a620f86e not found: ID does not exist" Sep 10 22:21:43.837730 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:43.837684 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg"] Sep 10 22:21:43.840058 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:43.840032 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2057d50e-75c69567c7-jcmgg"] Sep 10 22:21:44.937380 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:44.937344 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfc465df-6993-45e6-ae57-7262c4d0ab8b" path="/var/lib/kubelet/pods/dfc465df-6993-45e6-ae57-7262c4d0ab8b/volumes" Sep 10 22:21:50.669712 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:50.669681 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7"] Sep 10 22:21:50.670260 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:50.670240 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfc465df-6993-45e6-ae57-7262c4d0ab8b" containerName="maas-api" Sep 10 22:21:50.670310 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:50.670264 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfc465df-6993-45e6-ae57-7262c4d0ab8b" containerName="maas-api" Sep 10 22:21:50.670310 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:50.670283 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d11d98c9-655d-46f4-b5ce-51a850d64a25" containerName="maas-api" Sep 10 22:21:50.670310 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:50.670291 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d11d98c9-655d-46f4-b5ce-51a850d64a25" containerName="maas-api" Sep 10 22:21:50.670409 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:50.670392 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d11d98c9-655d-46f4-b5ce-51a850d64a25" containerName="maas-api" Sep 10 22:21:50.670409 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:50.670404 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfc465df-6993-45e6-ae57-7262c4d0ab8b" containerName="maas-api" Sep 10 22:21:50.674771 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:50.674752 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7" Sep 10 22:21:50.677255 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:50.677226 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-65b791-serving-cert\"" Sep 10 22:21:50.685107 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:50.685080 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7"] Sep 10 22:21:50.767674 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:50.767632 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6rbg\" (UniqueName: \"kubernetes.io/projected/b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed-kube-api-access-l6rbg\") pod \"maas-api-e2e-explicit-65b791-7856876c6-7bmp7\" (UID: \"b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7" Sep 10 22:21:50.767852 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:50.767700 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed-maas-api-tls\") pod \"maas-api-e2e-explicit-65b791-7856876c6-7bmp7\" (UID: \"b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7" Sep 10 22:21:50.767852 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:50.767761 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed-openshift-service-ca\") pod \"maas-api-e2e-explicit-65b791-7856876c6-7bmp7\" (UID: \"b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7" Sep 10 22:21:50.767852 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:50.767788 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed-metrics-certs\") pod \"maas-api-e2e-explicit-65b791-7856876c6-7bmp7\" (UID: \"b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7" Sep 10 22:21:50.868462 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:50.868423 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6rbg\" (UniqueName: \"kubernetes.io/projected/b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed-kube-api-access-l6rbg\") pod \"maas-api-e2e-explicit-65b791-7856876c6-7bmp7\" (UID: \"b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7" Sep 10 22:21:50.868655 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:50.868477 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed-maas-api-tls\") pod \"maas-api-e2e-explicit-65b791-7856876c6-7bmp7\" (UID: \"b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7" Sep 10 22:21:50.868655 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:50.868526 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed-openshift-service-ca\") pod \"maas-api-e2e-explicit-65b791-7856876c6-7bmp7\" (UID: \"b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7" Sep 10 22:21:50.868655 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:50.868549 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed-metrics-certs\") pod \"maas-api-e2e-explicit-65b791-7856876c6-7bmp7\" (UID: \"b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7" Sep 10 22:21:50.869227 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:50.869199 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed-openshift-service-ca\") pod \"maas-api-e2e-explicit-65b791-7856876c6-7bmp7\" (UID: \"b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7" Sep 10 22:21:50.871118 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:50.871094 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed-maas-api-tls\") pod \"maas-api-e2e-explicit-65b791-7856876c6-7bmp7\" (UID: \"b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7" Sep 10 22:21:50.871339 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:50.871322 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed-metrics-certs\") pod \"maas-api-e2e-explicit-65b791-7856876c6-7bmp7\" (UID: \"b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7" Sep 10 22:21:50.880573 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:50.880542 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6rbg\" (UniqueName: \"kubernetes.io/projected/b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed-kube-api-access-l6rbg\") pod \"maas-api-e2e-explicit-65b791-7856876c6-7bmp7\" (UID: \"b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7" Sep 10 22:21:50.987730 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:50.987633 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7" Sep 10 22:21:51.267498 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:51.267418 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7"] Sep 10 22:21:51.269826 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:21:51.269794 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22c9ca8_994b_4c52_a5f8_3f368b7ec6ed.slice/crio-efe8e047a542384f9e3462b4465c107d0fdde035c4e6f0c5c708706f291c4133 WatchSource:0}: Error finding container efe8e047a542384f9e3462b4465c107d0fdde035c4e6f0c5c708706f291c4133: Status 404 returned error can't find the container with id efe8e047a542384f9e3462b4465c107d0fdde035c4e6f0c5c708706f291c4133 Sep 10 22:21:51.271168 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:51.271150 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 22:21:51.847695 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:51.847646 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7" event={"ID":"b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed","Type":"ContainerStarted","Data":"27ee2418f44583442744a173887738debd66944991655000631624add726628c"} Sep 10 22:21:51.848188 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:51.847751 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7" event={"ID":"b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed","Type":"ContainerStarted","Data":"efe8e047a542384f9e3462b4465c107d0fdde035c4e6f0c5c708706f291c4133"} Sep 10 22:21:51.848188 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:51.847934 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7" Sep 10 22:21:51.872364 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:51.872306 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7" podStartSLOduration=1.424475274 podStartE2EDuration="1.872289969s" totalImagesPullingTime="447.814695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:21:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:21:51.870929462 +0000 UTC m=+1013.440419424" watchObservedRunningTime="2026-09-10 22:21:51.872289969 +0000 UTC m=+1013.441779931" Sep 10 22:21:57.858331 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:21:57.858300 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7" Sep 10 22:22:57.242584 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:22:57.242498 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7"] Sep 10 22:22:57.243119 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:22:57.242758 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7" podUID="b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed" containerName="maas-api" containerID="cri-o://27ee2418f44583442744a173887738debd66944991655000631624add726628c" gracePeriod=30 Sep 10 22:22:57.492388 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:22:57.492362 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7" Sep 10 22:22:57.570706 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:22:57.570630 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed-metrics-certs\") pod \"b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed\" (UID: \"b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed\") " Sep 10 22:22:57.570706 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:22:57.570676 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6rbg\" (UniqueName: \"kubernetes.io/projected/b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed-kube-api-access-l6rbg\") pod \"b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed\" (UID: \"b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed\") " Sep 10 22:22:57.570936 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:22:57.570742 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed-openshift-service-ca\") pod \"b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed\" (UID: \"b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed\") " Sep 10 22:22:57.570936 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:22:57.570757 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed-maas-api-tls\") pod \"b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed\" (UID: \"b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed\") " Sep 10 22:22:57.571144 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:22:57.571112 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed" (UID: "b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:22:57.572875 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:22:57.572850 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed" (UID: "b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:22:57.572970 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:22:57.572871 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed" (UID: "b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:22:57.572970 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:22:57.572888 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed-kube-api-access-l6rbg" (OuterVolumeSpecName: "kube-api-access-l6rbg") pod "b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed" (UID: "b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed"). InnerVolumeSpecName "kube-api-access-l6rbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:22:57.672487 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:22:57.672454 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed-openshift-service-ca\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:22:57.672487 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:22:57.672482 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed-maas-api-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:22:57.672487 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:22:57.672491 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed-metrics-certs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:22:57.672703 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:22:57.672498 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6rbg\" (UniqueName: \"kubernetes.io/projected/b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed-kube-api-access-l6rbg\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:22:58.169160 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:22:58.169124 2564 generic.go:356] "Generic (PLEG): container finished" podID="b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed" containerID="27ee2418f44583442744a173887738debd66944991655000631624add726628c" exitCode=0 Sep 10 22:22:58.169335 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:22:58.169165 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7" Sep 10 22:22:58.169335 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:22:58.169212 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7" event={"ID":"b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed","Type":"ContainerDied","Data":"27ee2418f44583442744a173887738debd66944991655000631624add726628c"} Sep 10 22:22:58.169335 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:22:58.169261 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7" event={"ID":"b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed","Type":"ContainerDied","Data":"efe8e047a542384f9e3462b4465c107d0fdde035c4e6f0c5c708706f291c4133"} Sep 10 22:22:58.169335 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:22:58.169289 2564 scope.go:117] "RemoveContainer" containerID="27ee2418f44583442744a173887738debd66944991655000631624add726628c" Sep 10 22:22:58.178633 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:22:58.178616 2564 scope.go:117] "RemoveContainer" containerID="27ee2418f44583442744a173887738debd66944991655000631624add726628c" Sep 10 22:22:58.178895 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:22:58.178881 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27ee2418f44583442744a173887738debd66944991655000631624add726628c\": container with ID starting with 27ee2418f44583442744a173887738debd66944991655000631624add726628c not found: ID does not exist" containerID="27ee2418f44583442744a173887738debd66944991655000631624add726628c" Sep 10 22:22:58.178990 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:22:58.178917 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27ee2418f44583442744a173887738debd66944991655000631624add726628c"} err="failed to get container status \"27ee2418f44583442744a173887738debd66944991655000631624add726628c\": rpc error: code = NotFound desc = could not find container \"27ee2418f44583442744a173887738debd66944991655000631624add726628c\": container with ID starting with 27ee2418f44583442744a173887738debd66944991655000631624add726628c not found: ID does not exist" Sep 10 22:22:58.189889 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:22:58.189867 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7"] Sep 10 22:22:58.196118 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:22:58.196086 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-65b791-7856876c6-7bmp7"] Sep 10 22:22:58.938536 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:22:58.938507 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed" path="/var/lib/kubelet/pods/b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed/volumes" Sep 10 22:23:23.301053 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:23.301023 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7"] Sep 10 22:23:23.301446 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:23.301433 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed" containerName="maas-api" Sep 10 22:23:23.301505 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:23.301447 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed" containerName="maas-api" Sep 10 22:23:23.301567 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:23.301517 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b22c9ca8-994b-4c52-a5f8-3f368b7ec6ed" containerName="maas-api" Sep 10 22:23:23.304261 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:23.304244 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7" Sep 10 22:23:23.306845 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:23.306799 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b2c924c3-serving-cert\"" Sep 10 22:23:23.313133 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:23.313109 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c392bc0-dc14-48d3-a5ec-47f0b8feed4d-metrics-certs\") pod \"maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7\" (UID: \"8c392bc0-dc14-48d3-a5ec-47f0b8feed4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7" Sep 10 22:23:23.313416 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:23.313388 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c392bc0-dc14-48d3-a5ec-47f0b8feed4d-openshift-service-ca\") pod \"maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7\" (UID: \"8c392bc0-dc14-48d3-a5ec-47f0b8feed4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7" Sep 10 22:23:23.313566 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:23.313549 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnj79\" (UniqueName: \"kubernetes.io/projected/8c392bc0-dc14-48d3-a5ec-47f0b8feed4d-kube-api-access-bnj79\") pod \"maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7\" (UID: \"8c392bc0-dc14-48d3-a5ec-47f0b8feed4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7" Sep 10 22:23:23.313639 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:23.313592 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c392bc0-dc14-48d3-a5ec-47f0b8feed4d-maas-api-tls\") pod \"maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7\" (UID: \"8c392bc0-dc14-48d3-a5ec-47f0b8feed4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7" Sep 10 22:23:23.315799 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:23.315773 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7"] Sep 10 22:23:23.414518 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:23.414478 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnj79\" (UniqueName: \"kubernetes.io/projected/8c392bc0-dc14-48d3-a5ec-47f0b8feed4d-kube-api-access-bnj79\") pod \"maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7\" (UID: \"8c392bc0-dc14-48d3-a5ec-47f0b8feed4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7" Sep 10 22:23:23.414518 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:23.414516 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c392bc0-dc14-48d3-a5ec-47f0b8feed4d-maas-api-tls\") pod \"maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7\" (UID: \"8c392bc0-dc14-48d3-a5ec-47f0b8feed4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7" Sep 10 22:23:23.414749 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:23.414550 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c392bc0-dc14-48d3-a5ec-47f0b8feed4d-metrics-certs\") pod \"maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7\" (UID: \"8c392bc0-dc14-48d3-a5ec-47f0b8feed4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7" Sep 10 22:23:23.414749 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:23.414593 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c392bc0-dc14-48d3-a5ec-47f0b8feed4d-openshift-service-ca\") pod \"maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7\" (UID: \"8c392bc0-dc14-48d3-a5ec-47f0b8feed4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7" Sep 10 22:23:23.415354 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:23.415330 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c392bc0-dc14-48d3-a5ec-47f0b8feed4d-openshift-service-ca\") pod \"maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7\" (UID: \"8c392bc0-dc14-48d3-a5ec-47f0b8feed4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7" Sep 10 22:23:23.417057 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:23.417033 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c392bc0-dc14-48d3-a5ec-47f0b8feed4d-metrics-certs\") pod \"maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7\" (UID: \"8c392bc0-dc14-48d3-a5ec-47f0b8feed4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7" Sep 10 22:23:23.417166 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:23.417072 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c392bc0-dc14-48d3-a5ec-47f0b8feed4d-maas-api-tls\") pod \"maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7\" (UID: \"8c392bc0-dc14-48d3-a5ec-47f0b8feed4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7" Sep 10 22:23:23.424006 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:23.423983 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnj79\" (UniqueName: \"kubernetes.io/projected/8c392bc0-dc14-48d3-a5ec-47f0b8feed4d-kube-api-access-bnj79\") pod \"maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7\" (UID: \"8c392bc0-dc14-48d3-a5ec-47f0b8feed4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7" Sep 10 22:23:23.615284 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:23.615250 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7" Sep 10 22:23:23.891138 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:23.891112 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7"] Sep 10 22:23:23.893373 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:23:23.893338 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c392bc0_dc14_48d3_a5ec_47f0b8feed4d.slice/crio-37beeecdad357d38daa7e4604e8341a2792e75fb5338581646af7252ba41c13a WatchSource:0}: Error finding container 37beeecdad357d38daa7e4604e8341a2792e75fb5338581646af7252ba41c13a: Status 404 returned error can't find the container with id 37beeecdad357d38daa7e4604e8341a2792e75fb5338581646af7252ba41c13a Sep 10 22:23:24.296860 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:24.296774 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7" event={"ID":"8c392bc0-dc14-48d3-a5ec-47f0b8feed4d","Type":"ContainerStarted","Data":"37beeecdad357d38daa7e4604e8341a2792e75fb5338581646af7252ba41c13a"} Sep 10 22:23:25.303497 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:25.303460 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7" event={"ID":"8c392bc0-dc14-48d3-a5ec-47f0b8feed4d","Type":"ContainerStarted","Data":"e045fb512837507fc4d346ec0fe5717f2195e7aae5f89efc29c805ac5500a9b4"} Sep 10 22:23:25.303935 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:25.303596 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7" Sep 10 22:23:25.320854 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:25.320805 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7" podStartSLOduration=1.5893508330000001 podStartE2EDuration="2.320790705s" totalImagesPullingTime="731.439872ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:23:25.319460922 +0000 UTC m=+1106.888950892" watchObservedRunningTime="2026-09-10 22:23:25.320790705 +0000 UTC m=+1106.890280669" Sep 10 22:23:31.313979 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:31.313953 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7" Sep 10 22:23:48.773826 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:48.773784 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g"] Sep 10 22:23:48.774350 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:48.774016 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g" podUID="371a2114-dc48-4bfb-82c2-a44550f9cf67" containerName="maas-api" containerID="cri-o://ce3c541039cbe389410727fcf0a09d38ea352d4600ce5e10a886a91ccac66277" gracePeriod=30 Sep 10 22:23:49.234473 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:49.234451 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g" Sep 10 22:23:49.378210 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:49.378184 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n56nb\" (UniqueName: \"kubernetes.io/projected/371a2114-dc48-4bfb-82c2-a44550f9cf67-kube-api-access-n56nb\") pod \"371a2114-dc48-4bfb-82c2-a44550f9cf67\" (UID: \"371a2114-dc48-4bfb-82c2-a44550f9cf67\") " Sep 10 22:23:49.378369 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:49.378236 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/371a2114-dc48-4bfb-82c2-a44550f9cf67-maas-api-tls\") pod \"371a2114-dc48-4bfb-82c2-a44550f9cf67\" (UID: \"371a2114-dc48-4bfb-82c2-a44550f9cf67\") " Sep 10 22:23:49.378369 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:49.378253 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371a2114-dc48-4bfb-82c2-a44550f9cf67-metrics-certs\") pod \"371a2114-dc48-4bfb-82c2-a44550f9cf67\" (UID: \"371a2114-dc48-4bfb-82c2-a44550f9cf67\") " Sep 10 22:23:49.378369 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:49.378349 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/371a2114-dc48-4bfb-82c2-a44550f9cf67-openshift-service-ca\") pod \"371a2114-dc48-4bfb-82c2-a44550f9cf67\" (UID: \"371a2114-dc48-4bfb-82c2-a44550f9cf67\") " Sep 10 22:23:49.378720 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:49.378691 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371a2114-dc48-4bfb-82c2-a44550f9cf67-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "371a2114-dc48-4bfb-82c2-a44550f9cf67" (UID: "371a2114-dc48-4bfb-82c2-a44550f9cf67"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:23:49.380313 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:49.380290 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/371a2114-dc48-4bfb-82c2-a44550f9cf67-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "371a2114-dc48-4bfb-82c2-a44550f9cf67" (UID: "371a2114-dc48-4bfb-82c2-a44550f9cf67"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:23:49.380406 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:49.380334 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/371a2114-dc48-4bfb-82c2-a44550f9cf67-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "371a2114-dc48-4bfb-82c2-a44550f9cf67" (UID: "371a2114-dc48-4bfb-82c2-a44550f9cf67"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:23:49.380406 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:49.380343 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371a2114-dc48-4bfb-82c2-a44550f9cf67-kube-api-access-n56nb" (OuterVolumeSpecName: "kube-api-access-n56nb") pod "371a2114-dc48-4bfb-82c2-a44550f9cf67" (UID: "371a2114-dc48-4bfb-82c2-a44550f9cf67"). InnerVolumeSpecName "kube-api-access-n56nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:23:49.427843 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:49.427814 2564 generic.go:356] "Generic (PLEG): container finished" podID="371a2114-dc48-4bfb-82c2-a44550f9cf67" containerID="ce3c541039cbe389410727fcf0a09d38ea352d4600ce5e10a886a91ccac66277" exitCode=0 Sep 10 22:23:49.427980 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:49.427857 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g" Sep 10 22:23:49.427980 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:49.427920 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g" event={"ID":"371a2114-dc48-4bfb-82c2-a44550f9cf67","Type":"ContainerDied","Data":"ce3c541039cbe389410727fcf0a09d38ea352d4600ce5e10a886a91ccac66277"} Sep 10 22:23:49.427980 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:49.427968 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g" event={"ID":"371a2114-dc48-4bfb-82c2-a44550f9cf67","Type":"ContainerDied","Data":"b33be3755c7a3e6bf11440d315a141e8f51d9f3545187133c7f2d680f44b87f6"} Sep 10 22:23:49.428093 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:49.427991 2564 scope.go:117] "RemoveContainer" containerID="ce3c541039cbe389410727fcf0a09d38ea352d4600ce5e10a886a91ccac66277" Sep 10 22:23:49.436840 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:49.436825 2564 scope.go:117] "RemoveContainer" containerID="ce3c541039cbe389410727fcf0a09d38ea352d4600ce5e10a886a91ccac66277" Sep 10 22:23:49.437103 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:23:49.437086 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce3c541039cbe389410727fcf0a09d38ea352d4600ce5e10a886a91ccac66277\": container with ID starting with ce3c541039cbe389410727fcf0a09d38ea352d4600ce5e10a886a91ccac66277 not found: ID does not exist" containerID="ce3c541039cbe389410727fcf0a09d38ea352d4600ce5e10a886a91ccac66277" Sep 10 22:23:49.437147 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:49.437113 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce3c541039cbe389410727fcf0a09d38ea352d4600ce5e10a886a91ccac66277"} err="failed to get container status \"ce3c541039cbe389410727fcf0a09d38ea352d4600ce5e10a886a91ccac66277\": rpc error: code = NotFound desc = could not find container \"ce3c541039cbe389410727fcf0a09d38ea352d4600ce5e10a886a91ccac66277\": container with ID starting with ce3c541039cbe389410727fcf0a09d38ea352d4600ce5e10a886a91ccac66277 not found: ID does not exist" Sep 10 22:23:49.461580 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:49.461558 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g"] Sep 10 22:23:49.470300 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:49.470280 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-669e2b-85f8dc5777-9fp9g"] Sep 10 22:23:49.480001 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:49.479984 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n56nb\" (UniqueName: \"kubernetes.io/projected/371a2114-dc48-4bfb-82c2-a44550f9cf67-kube-api-access-n56nb\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:23:49.480001 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:49.480001 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/371a2114-dc48-4bfb-82c2-a44550f9cf67-maas-api-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:23:49.480127 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:49.480010 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371a2114-dc48-4bfb-82c2-a44550f9cf67-metrics-certs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:23:49.480127 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:49.480019 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/371a2114-dc48-4bfb-82c2-a44550f9cf67-openshift-service-ca\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:23:50.937510 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:23:50.937476 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="371a2114-dc48-4bfb-82c2-a44550f9cf67" path="/var/lib/kubelet/pods/371a2114-dc48-4bfb-82c2-a44550f9cf67/volumes" Sep 10 22:24:05.095066 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:05.095027 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7"] Sep 10 22:24:05.095631 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:05.095238 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7" podUID="8c392bc0-dc14-48d3-a5ec-47f0b8feed4d" containerName="maas-api" containerID="cri-o://e045fb512837507fc4d346ec0fe5717f2195e7aae5f89efc29c805ac5500a9b4" gracePeriod=30 Sep 10 22:24:05.349691 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:05.349632 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7" Sep 10 22:24:05.428142 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:05.428110 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c392bc0-dc14-48d3-a5ec-47f0b8feed4d-metrics-certs\") pod \"8c392bc0-dc14-48d3-a5ec-47f0b8feed4d\" (UID: \"8c392bc0-dc14-48d3-a5ec-47f0b8feed4d\") " Sep 10 22:24:05.428311 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:05.428150 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c392bc0-dc14-48d3-a5ec-47f0b8feed4d-openshift-service-ca\") pod \"8c392bc0-dc14-48d3-a5ec-47f0b8feed4d\" (UID: \"8c392bc0-dc14-48d3-a5ec-47f0b8feed4d\") " Sep 10 22:24:05.428311 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:05.428174 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnj79\" (UniqueName: \"kubernetes.io/projected/8c392bc0-dc14-48d3-a5ec-47f0b8feed4d-kube-api-access-bnj79\") pod \"8c392bc0-dc14-48d3-a5ec-47f0b8feed4d\" (UID: \"8c392bc0-dc14-48d3-a5ec-47f0b8feed4d\") " Sep 10 22:24:05.428311 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:05.428231 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c392bc0-dc14-48d3-a5ec-47f0b8feed4d-maas-api-tls\") pod \"8c392bc0-dc14-48d3-a5ec-47f0b8feed4d\" (UID: \"8c392bc0-dc14-48d3-a5ec-47f0b8feed4d\") " Sep 10 22:24:05.428721 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:05.428689 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c392bc0-dc14-48d3-a5ec-47f0b8feed4d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8c392bc0-dc14-48d3-a5ec-47f0b8feed4d" (UID: "8c392bc0-dc14-48d3-a5ec-47f0b8feed4d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:24:05.430285 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:05.430245 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c392bc0-dc14-48d3-a5ec-47f0b8feed4d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8c392bc0-dc14-48d3-a5ec-47f0b8feed4d" (UID: "8c392bc0-dc14-48d3-a5ec-47f0b8feed4d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:24:05.430664 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:05.430637 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c392bc0-dc14-48d3-a5ec-47f0b8feed4d-kube-api-access-bnj79" (OuterVolumeSpecName: "kube-api-access-bnj79") pod "8c392bc0-dc14-48d3-a5ec-47f0b8feed4d" (UID: "8c392bc0-dc14-48d3-a5ec-47f0b8feed4d"). InnerVolumeSpecName "kube-api-access-bnj79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:24:05.430806 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:05.430675 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c392bc0-dc14-48d3-a5ec-47f0b8feed4d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8c392bc0-dc14-48d3-a5ec-47f0b8feed4d" (UID: "8c392bc0-dc14-48d3-a5ec-47f0b8feed4d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:24:05.507195 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:05.507166 2564 generic.go:356] "Generic (PLEG): container finished" podID="8c392bc0-dc14-48d3-a5ec-47f0b8feed4d" containerID="e045fb512837507fc4d346ec0fe5717f2195e7aae5f89efc29c805ac5500a9b4" exitCode=0 Sep 10 22:24:05.507307 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:05.507210 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7" Sep 10 22:24:05.507307 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:05.507217 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7" event={"ID":"8c392bc0-dc14-48d3-a5ec-47f0b8feed4d","Type":"ContainerDied","Data":"e045fb512837507fc4d346ec0fe5717f2195e7aae5f89efc29c805ac5500a9b4"} Sep 10 22:24:05.507307 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:05.507262 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7" event={"ID":"8c392bc0-dc14-48d3-a5ec-47f0b8feed4d","Type":"ContainerDied","Data":"37beeecdad357d38daa7e4604e8341a2792e75fb5338581646af7252ba41c13a"} Sep 10 22:24:05.507307 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:05.507284 2564 scope.go:117] "RemoveContainer" containerID="e045fb512837507fc4d346ec0fe5717f2195e7aae5f89efc29c805ac5500a9b4" Sep 10 22:24:05.516997 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:05.516979 2564 scope.go:117] "RemoveContainer" containerID="e045fb512837507fc4d346ec0fe5717f2195e7aae5f89efc29c805ac5500a9b4" Sep 10 22:24:05.517253 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:24:05.517234 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e045fb512837507fc4d346ec0fe5717f2195e7aae5f89efc29c805ac5500a9b4\": container with ID starting with e045fb512837507fc4d346ec0fe5717f2195e7aae5f89efc29c805ac5500a9b4 not found: ID does not exist" containerID="e045fb512837507fc4d346ec0fe5717f2195e7aae5f89efc29c805ac5500a9b4" Sep 10 22:24:05.517311 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:05.517262 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e045fb512837507fc4d346ec0fe5717f2195e7aae5f89efc29c805ac5500a9b4"} err="failed to get container status \"e045fb512837507fc4d346ec0fe5717f2195e7aae5f89efc29c805ac5500a9b4\": rpc error: code = NotFound desc = could not find container \"e045fb512837507fc4d346ec0fe5717f2195e7aae5f89efc29c805ac5500a9b4\": container with ID starting with e045fb512837507fc4d346ec0fe5717f2195e7aae5f89efc29c805ac5500a9b4 not found: ID does not exist" Sep 10 22:24:05.528626 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:05.528603 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7"] Sep 10 22:24:05.529043 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:05.529024 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c392bc0-dc14-48d3-a5ec-47f0b8feed4d-maas-api-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:24:05.529119 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:05.529047 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c392bc0-dc14-48d3-a5ec-47f0b8feed4d-metrics-certs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:24:05.529119 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:05.529060 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c392bc0-dc14-48d3-a5ec-47f0b8feed4d-openshift-service-ca\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:24:05.529119 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:05.529072 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnj79\" (UniqueName: \"kubernetes.io/projected/8c392bc0-dc14-48d3-a5ec-47f0b8feed4d-kube-api-access-bnj79\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:24:05.533028 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:05.533007 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b2c924c3-5584b5958f-pw4j7"] Sep 10 22:24:06.675394 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:06.675364 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5"] Sep 10 22:24:06.675839 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:06.675600 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" podUID="82cb99cc-1e95-43ee-8209-f9113ed0e5ea" containerName="maas-api" containerID="cri-o://aee7de5cad0ccf2a4ece708d176e0e5bbd3d03443a8ec74d9e034cadecf82962" gracePeriod=30 Sep 10 22:24:06.931454 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:06.931397 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" Sep 10 22:24:06.937263 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:06.937239 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c392bc0-dc14-48d3-a5ec-47f0b8feed4d" path="/var/lib/kubelet/pods/8c392bc0-dc14-48d3-a5ec-47f0b8feed4d/volumes" Sep 10 22:24:07.041888 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:07.041863 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-metrics-certs\") pod \"82cb99cc-1e95-43ee-8209-f9113ed0e5ea\" (UID: \"82cb99cc-1e95-43ee-8209-f9113ed0e5ea\") " Sep 10 22:24:07.042044 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:07.041896 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-maas-api-tls\") pod \"82cb99cc-1e95-43ee-8209-f9113ed0e5ea\" (UID: \"82cb99cc-1e95-43ee-8209-f9113ed0e5ea\") " Sep 10 22:24:07.042044 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:07.041991 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77s2l\" (UniqueName: \"kubernetes.io/projected/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-kube-api-access-77s2l\") pod \"82cb99cc-1e95-43ee-8209-f9113ed0e5ea\" (UID: \"82cb99cc-1e95-43ee-8209-f9113ed0e5ea\") " Sep 10 22:24:07.042044 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:07.042011 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-openshift-service-ca\") pod \"82cb99cc-1e95-43ee-8209-f9113ed0e5ea\" (UID: \"82cb99cc-1e95-43ee-8209-f9113ed0e5ea\") " Sep 10 22:24:07.042456 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:07.042413 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "82cb99cc-1e95-43ee-8209-f9113ed0e5ea" (UID: "82cb99cc-1e95-43ee-8209-f9113ed0e5ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:24:07.043973 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:07.043946 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "82cb99cc-1e95-43ee-8209-f9113ed0e5ea" (UID: "82cb99cc-1e95-43ee-8209-f9113ed0e5ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:24:07.044402 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:07.044382 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-kube-api-access-77s2l" (OuterVolumeSpecName: "kube-api-access-77s2l") pod "82cb99cc-1e95-43ee-8209-f9113ed0e5ea" (UID: "82cb99cc-1e95-43ee-8209-f9113ed0e5ea"). InnerVolumeSpecName "kube-api-access-77s2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:24:07.044402 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:07.044382 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "82cb99cc-1e95-43ee-8209-f9113ed0e5ea" (UID: "82cb99cc-1e95-43ee-8209-f9113ed0e5ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:24:07.143390 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:07.143350 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-metrics-certs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:24:07.143390 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:07.143378 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-maas-api-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:24:07.143390 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:07.143392 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77s2l\" (UniqueName: \"kubernetes.io/projected/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-kube-api-access-77s2l\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:24:07.143602 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:07.143403 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82cb99cc-1e95-43ee-8209-f9113ed0e5ea-openshift-service-ca\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:24:07.520177 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:07.520142 2564 generic.go:356] "Generic (PLEG): container finished" podID="82cb99cc-1e95-43ee-8209-f9113ed0e5ea" containerID="aee7de5cad0ccf2a4ece708d176e0e5bbd3d03443a8ec74d9e034cadecf82962" exitCode=0 Sep 10 22:24:07.520177 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:07.520180 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" Sep 10 22:24:07.520369 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:07.520222 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" event={"ID":"82cb99cc-1e95-43ee-8209-f9113ed0e5ea","Type":"ContainerDied","Data":"aee7de5cad0ccf2a4ece708d176e0e5bbd3d03443a8ec74d9e034cadecf82962"} Sep 10 22:24:07.520369 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:07.520275 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5" event={"ID":"82cb99cc-1e95-43ee-8209-f9113ed0e5ea","Type":"ContainerDied","Data":"9a92b8773b704f5d506f77cc4b4f704cd61a6c1ca68cba9743772731a6db8a31"} Sep 10 22:24:07.520369 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:07.520299 2564 scope.go:117] "RemoveContainer" containerID="aee7de5cad0ccf2a4ece708d176e0e5bbd3d03443a8ec74d9e034cadecf82962" Sep 10 22:24:07.529524 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:07.529504 2564 scope.go:117] "RemoveContainer" containerID="aee7de5cad0ccf2a4ece708d176e0e5bbd3d03443a8ec74d9e034cadecf82962" Sep 10 22:24:07.529787 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:24:07.529770 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aee7de5cad0ccf2a4ece708d176e0e5bbd3d03443a8ec74d9e034cadecf82962\": container with ID starting with aee7de5cad0ccf2a4ece708d176e0e5bbd3d03443a8ec74d9e034cadecf82962 not found: ID does not exist" containerID="aee7de5cad0ccf2a4ece708d176e0e5bbd3d03443a8ec74d9e034cadecf82962" Sep 10 22:24:07.529832 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:07.529797 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aee7de5cad0ccf2a4ece708d176e0e5bbd3d03443a8ec74d9e034cadecf82962"} err="failed to get container status \"aee7de5cad0ccf2a4ece708d176e0e5bbd3d03443a8ec74d9e034cadecf82962\": rpc error: code = NotFound desc = could not find container \"aee7de5cad0ccf2a4ece708d176e0e5bbd3d03443a8ec74d9e034cadecf82962\": container with ID starting with aee7de5cad0ccf2a4ece708d176e0e5bbd3d03443a8ec74d9e034cadecf82962 not found: ID does not exist" Sep 10 22:24:07.542930 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:07.541721 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5"] Sep 10 22:24:07.550113 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:07.550090 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c3af05-6454f5d49b-79rk5"] Sep 10 22:24:08.937214 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:08.937181 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82cb99cc-1e95-43ee-8209-f9113ed0e5ea" path="/var/lib/kubelet/pods/82cb99cc-1e95-43ee-8209-f9113ed0e5ea/volumes" Sep 10 22:24:18.501258 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:18.501228 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw"] Sep 10 22:24:18.501632 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:18.501615 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c392bc0-dc14-48d3-a5ec-47f0b8feed4d" containerName="maas-api" Sep 10 22:24:18.501632 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:18.501627 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c392bc0-dc14-48d3-a5ec-47f0b8feed4d" containerName="maas-api" Sep 10 22:24:18.501714 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:18.501639 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="371a2114-dc48-4bfb-82c2-a44550f9cf67" containerName="maas-api" Sep 10 22:24:18.501714 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:18.501644 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="371a2114-dc48-4bfb-82c2-a44550f9cf67" containerName="maas-api" Sep 10 22:24:18.501714 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:18.501658 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82cb99cc-1e95-43ee-8209-f9113ed0e5ea" containerName="maas-api" Sep 10 22:24:18.501714 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:18.501663 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="82cb99cc-1e95-43ee-8209-f9113ed0e5ea" containerName="maas-api" Sep 10 22:24:18.501848 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:18.501727 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c392bc0-dc14-48d3-a5ec-47f0b8feed4d" containerName="maas-api" Sep 10 22:24:18.501848 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:18.501737 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="371a2114-dc48-4bfb-82c2-a44550f9cf67" containerName="maas-api" Sep 10 22:24:18.501848 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:18.501743 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="82cb99cc-1e95-43ee-8209-f9113ed0e5ea" containerName="maas-api" Sep 10 22:24:18.505288 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:18.505267 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" Sep 10 22:24:18.507730 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:18.507709 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-e367c1-serving-cert\"" Sep 10 22:24:18.517360 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:18.517333 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw"] Sep 10 22:24:18.644950 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:18.644888 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2flc7\" (UniqueName: \"kubernetes.io/projected/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-kube-api-access-2flc7\") pod \"maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw\" (UID: \"8de8a0cd-6f57-4778-b376-b64ca7e91d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" Sep 10 22:24:18.644950 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:18.644956 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw\" (UID: \"8de8a0cd-6f57-4778-b376-b64ca7e91d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" Sep 10 22:24:18.645178 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:18.644990 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw\" (UID: \"8de8a0cd-6f57-4778-b376-b64ca7e91d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" Sep 10 22:24:18.645178 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:18.645013 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw\" (UID: \"8de8a0cd-6f57-4778-b376-b64ca7e91d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" Sep 10 22:24:18.746410 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:18.746377 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2flc7\" (UniqueName: \"kubernetes.io/projected/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-kube-api-access-2flc7\") pod \"maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw\" (UID: \"8de8a0cd-6f57-4778-b376-b64ca7e91d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" Sep 10 22:24:18.746582 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:18.746422 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw\" (UID: \"8de8a0cd-6f57-4778-b376-b64ca7e91d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" Sep 10 22:24:18.746582 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:18.746451 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw\" (UID: \"8de8a0cd-6f57-4778-b376-b64ca7e91d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" Sep 10 22:24:18.746582 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:18.746476 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw\" (UID: \"8de8a0cd-6f57-4778-b376-b64ca7e91d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" Sep 10 22:24:18.746701 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:24:18.746610 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-serving-cert: secret "maas-api-e2e-shared-a-w3-e367c1-serving-cert" not found Sep 10 22:24:18.746701 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:24:18.746688 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-maas-api-tls podName:8de8a0cd-6f57-4778-b376-b64ca7e91d0c nodeName:}" failed. No retries permitted until 2026-09-10 22:24:19.24666696 +0000 UTC m=+1160.816156921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-maas-api-tls") pod "maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" (UID: "8de8a0cd-6f57-4778-b376-b64ca7e91d0c") : secret "maas-api-e2e-shared-a-w3-e367c1-serving-cert" not found Sep 10 22:24:18.747066 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:18.747043 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw\" (UID: \"8de8a0cd-6f57-4778-b376-b64ca7e91d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" Sep 10 22:24:18.748950 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:18.748929 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw\" (UID: \"8de8a0cd-6f57-4778-b376-b64ca7e91d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" Sep 10 22:24:18.754658 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:18.754602 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2flc7\" (UniqueName: \"kubernetes.io/projected/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-kube-api-access-2flc7\") pod \"maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw\" (UID: \"8de8a0cd-6f57-4778-b376-b64ca7e91d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" Sep 10 22:24:19.252264 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:19.252231 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw\" (UID: \"8de8a0cd-6f57-4778-b376-b64ca7e91d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" Sep 10 22:24:19.254817 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:19.254793 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw\" (UID: \"8de8a0cd-6f57-4778-b376-b64ca7e91d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" Sep 10 22:24:19.416553 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:19.416512 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" Sep 10 22:24:19.694299 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:19.694272 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw"] Sep 10 22:24:19.696498 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:24:19.696467 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de8a0cd_6f57_4778_b376_b64ca7e91d0c.slice/crio-44741d41ce7488c25c7d8beaab228f9d4bfcc7b749de39bd02151e6b62099a8f WatchSource:0}: Error finding container 44741d41ce7488c25c7d8beaab228f9d4bfcc7b749de39bd02151e6b62099a8f: Status 404 returned error can't find the container with id 44741d41ce7488c25c7d8beaab228f9d4bfcc7b749de39bd02151e6b62099a8f Sep 10 22:24:20.595879 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:20.595838 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" event={"ID":"8de8a0cd-6f57-4778-b376-b64ca7e91d0c","Type":"ContainerStarted","Data":"766c3ea70814915d15ca0ad545e1e2034ffc542b7a16b2ddc28427f97af967f6"} Sep 10 22:24:20.596073 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:20.595893 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" event={"ID":"8de8a0cd-6f57-4778-b376-b64ca7e91d0c","Type":"ContainerStarted","Data":"44741d41ce7488c25c7d8beaab228f9d4bfcc7b749de39bd02151e6b62099a8f"} Sep 10 22:24:20.596073 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:20.595950 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" Sep 10 22:24:20.620322 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:20.620267 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" podStartSLOduration=2.239828764 podStartE2EDuration="2.620251605s" totalImagesPullingTime="380.422841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:24:20.618135166 +0000 UTC m=+1162.187625129" watchObservedRunningTime="2026-09-10 22:24:20.620251605 +0000 UTC m=+1162.189741570" Sep 10 22:24:26.605270 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:26.605182 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" Sep 10 22:24:29.720142 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:29.720107 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg"] Sep 10 22:24:29.723707 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:29.723685 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg" Sep 10 22:24:29.726743 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:29.726720 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-4a864c-serving-cert\"" Sep 10 22:24:29.734564 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:29.734538 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg"] Sep 10 22:24:29.858621 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:29.858590 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b644ebc-739f-451b-b8c0-b2fd6852d45a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg\" (UID: \"4b644ebc-739f-451b-b8c0-b2fd6852d45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg" Sep 10 22:24:29.858803 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:29.858636 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b644ebc-739f-451b-b8c0-b2fd6852d45a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg\" (UID: \"4b644ebc-739f-451b-b8c0-b2fd6852d45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg" Sep 10 22:24:29.858803 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:29.858665 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b644ebc-739f-451b-b8c0-b2fd6852d45a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg\" (UID: \"4b644ebc-739f-451b-b8c0-b2fd6852d45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg" Sep 10 22:24:29.858803 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:29.858787 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pchkc\" (UniqueName: \"kubernetes.io/projected/4b644ebc-739f-451b-b8c0-b2fd6852d45a-kube-api-access-pchkc\") pod \"maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg\" (UID: \"4b644ebc-739f-451b-b8c0-b2fd6852d45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg" Sep 10 22:24:29.960010 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:29.959978 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b644ebc-739f-451b-b8c0-b2fd6852d45a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg\" (UID: \"4b644ebc-739f-451b-b8c0-b2fd6852d45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg" Sep 10 22:24:29.960224 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:29.960021 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b644ebc-739f-451b-b8c0-b2fd6852d45a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg\" (UID: \"4b644ebc-739f-451b-b8c0-b2fd6852d45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg" Sep 10 22:24:29.960224 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:29.960088 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pchkc\" (UniqueName: \"kubernetes.io/projected/4b644ebc-739f-451b-b8c0-b2fd6852d45a-kube-api-access-pchkc\") pod \"maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg\" (UID: \"4b644ebc-739f-451b-b8c0-b2fd6852d45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg" Sep 10 22:24:29.960224 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:29.960172 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b644ebc-739f-451b-b8c0-b2fd6852d45a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg\" (UID: \"4b644ebc-739f-451b-b8c0-b2fd6852d45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg" Sep 10 22:24:29.960952 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:29.960926 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b644ebc-739f-451b-b8c0-b2fd6852d45a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg\" (UID: \"4b644ebc-739f-451b-b8c0-b2fd6852d45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg" Sep 10 22:24:29.962752 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:29.962716 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b644ebc-739f-451b-b8c0-b2fd6852d45a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg\" (UID: \"4b644ebc-739f-451b-b8c0-b2fd6852d45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg" Sep 10 22:24:29.962914 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:29.962791 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b644ebc-739f-451b-b8c0-b2fd6852d45a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg\" (UID: \"4b644ebc-739f-451b-b8c0-b2fd6852d45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg" Sep 10 22:24:29.968702 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:29.968680 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pchkc\" (UniqueName: \"kubernetes.io/projected/4b644ebc-739f-451b-b8c0-b2fd6852d45a-kube-api-access-pchkc\") pod \"maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg\" (UID: \"4b644ebc-739f-451b-b8c0-b2fd6852d45a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg" Sep 10 22:24:30.036038 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:30.035963 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg" Sep 10 22:24:30.309919 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:30.309881 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg"] Sep 10 22:24:30.311556 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:24:30.311532 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b644ebc_739f_451b_b8c0_b2fd6852d45a.slice/crio-eaa0ddfe9502041d4480aaebca2a63f96b9db35db6ca4af33fb01a563e4131f0 WatchSource:0}: Error finding container eaa0ddfe9502041d4480aaebca2a63f96b9db35db6ca4af33fb01a563e4131f0: Status 404 returned error can't find the container with id eaa0ddfe9502041d4480aaebca2a63f96b9db35db6ca4af33fb01a563e4131f0 Sep 10 22:24:30.648101 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:30.648052 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg" event={"ID":"4b644ebc-739f-451b-b8c0-b2fd6852d45a","Type":"ContainerStarted","Data":"eaa0ddfe9502041d4480aaebca2a63f96b9db35db6ca4af33fb01a563e4131f0"} Sep 10 22:24:31.656342 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:31.656307 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg" event={"ID":"4b644ebc-739f-451b-b8c0-b2fd6852d45a","Type":"ContainerStarted","Data":"4d5643110ffc733b285082f4c54c7ded8ba6ac06f4a684db2721b429f8c022c2"} Sep 10 22:24:31.656761 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:31.656478 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg" Sep 10 22:24:31.675411 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:31.675358 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg" podStartSLOduration=2.267469858 podStartE2EDuration="2.675344425s" totalImagesPullingTime="407.874567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:24:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:24:31.67340259 +0000 UTC m=+1173.242892554" watchObservedRunningTime="2026-09-10 22:24:31.675344425 +0000 UTC m=+1173.244834389" Sep 10 22:24:37.666286 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:37.666252 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg" Sep 10 22:24:46.886848 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:46.886802 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8"] Sep 10 22:24:46.890915 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:46.890881 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8" Sep 10 22:24:46.893585 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:46.893565 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-0ada08-serving-cert\"" Sep 10 22:24:46.901720 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:46.901702 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8"] Sep 10 22:24:47.026074 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:47.026042 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcgkv\" (UniqueName: \"kubernetes.io/projected/ab5c1a77-830d-463c-a6c2-a1965ee9ffad-kube-api-access-rcgkv\") pod \"maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8\" (UID: \"ab5c1a77-830d-463c-a6c2-a1965ee9ffad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8" Sep 10 22:24:47.026245 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:47.026127 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5c1a77-830d-463c-a6c2-a1965ee9ffad-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8\" (UID: \"ab5c1a77-830d-463c-a6c2-a1965ee9ffad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8" Sep 10 22:24:47.026245 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:47.026217 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5c1a77-830d-463c-a6c2-a1965ee9ffad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8\" (UID: \"ab5c1a77-830d-463c-a6c2-a1965ee9ffad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8" Sep 10 22:24:47.026339 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:47.026255 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab5c1a77-830d-463c-a6c2-a1965ee9ffad-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8\" (UID: \"ab5c1a77-830d-463c-a6c2-a1965ee9ffad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8" Sep 10 22:24:47.127036 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:47.126999 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5c1a77-830d-463c-a6c2-a1965ee9ffad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8\" (UID: \"ab5c1a77-830d-463c-a6c2-a1965ee9ffad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8" Sep 10 22:24:47.127217 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:47.127047 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab5c1a77-830d-463c-a6c2-a1965ee9ffad-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8\" (UID: \"ab5c1a77-830d-463c-a6c2-a1965ee9ffad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8" Sep 10 22:24:47.127217 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:47.127080 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcgkv\" (UniqueName: \"kubernetes.io/projected/ab5c1a77-830d-463c-a6c2-a1965ee9ffad-kube-api-access-rcgkv\") pod \"maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8\" (UID: \"ab5c1a77-830d-463c-a6c2-a1965ee9ffad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8" Sep 10 22:24:47.127217 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:47.127136 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5c1a77-830d-463c-a6c2-a1965ee9ffad-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8\" (UID: \"ab5c1a77-830d-463c-a6c2-a1965ee9ffad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8" Sep 10 22:24:47.127828 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:47.127806 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5c1a77-830d-463c-a6c2-a1965ee9ffad-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8\" (UID: \"ab5c1a77-830d-463c-a6c2-a1965ee9ffad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8" Sep 10 22:24:47.129600 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:47.129573 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab5c1a77-830d-463c-a6c2-a1965ee9ffad-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8\" (UID: \"ab5c1a77-830d-463c-a6c2-a1965ee9ffad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8" Sep 10 22:24:47.129689 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:47.129588 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5c1a77-830d-463c-a6c2-a1965ee9ffad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8\" (UID: \"ab5c1a77-830d-463c-a6c2-a1965ee9ffad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8" Sep 10 22:24:47.135810 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:47.135793 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcgkv\" (UniqueName: \"kubernetes.io/projected/ab5c1a77-830d-463c-a6c2-a1965ee9ffad-kube-api-access-rcgkv\") pod \"maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8\" (UID: \"ab5c1a77-830d-463c-a6c2-a1965ee9ffad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8" Sep 10 22:24:47.202706 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:47.202650 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8" Sep 10 22:24:47.477773 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:47.477684 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8"] Sep 10 22:24:47.481954 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:24:47.481921 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5c1a77_830d_463c_a6c2_a1965ee9ffad.slice/crio-608e41df7ac3130054e2a33e72ce227c20e245c6aef8ababa7146efd7a9b25e9 WatchSource:0}: Error finding container 608e41df7ac3130054e2a33e72ce227c20e245c6aef8ababa7146efd7a9b25e9: Status 404 returned error can't find the container with id 608e41df7ac3130054e2a33e72ce227c20e245c6aef8ababa7146efd7a9b25e9 Sep 10 22:24:47.742371 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:47.742291 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8" event={"ID":"ab5c1a77-830d-463c-a6c2-a1965ee9ffad","Type":"ContainerStarted","Data":"608e41df7ac3130054e2a33e72ce227c20e245c6aef8ababa7146efd7a9b25e9"} Sep 10 22:24:48.749125 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:48.749092 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8" event={"ID":"ab5c1a77-830d-463c-a6c2-a1965ee9ffad","Type":"ContainerStarted","Data":"74064813a2bb82a1a01bb77cab76f999dbfade7dfdf05e92d5d8c2da448d3907"} Sep 10 22:24:48.749529 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:48.749180 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8" Sep 10 22:24:48.767174 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:48.767129 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8" podStartSLOduration=2.428682887 podStartE2EDuration="2.76711541s" totalImagesPullingTime="338.432523ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:24:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:24:48.766220091 +0000 UTC m=+1190.335710056" watchObservedRunningTime="2026-09-10 22:24:48.76711541 +0000 UTC m=+1190.336605373" Sep 10 22:24:54.759009 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:54.758977 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8" Sep 10 22:24:58.902597 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:58.902560 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc"] Sep 10 22:24:58.906122 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:58.906100 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" Sep 10 22:24:58.908684 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:58.908663 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-32343f-serving-cert\"" Sep 10 22:24:58.918306 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:58.918238 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc"] Sep 10 22:24:59.033735 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:59.033711 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fdbdc87-97f2-43ca-b3d6-099653368751-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc\" (UID: \"2fdbdc87-97f2-43ca-b3d6-099653368751\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" Sep 10 22:24:59.033817 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:59.033751 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbpfv\" (UniqueName: \"kubernetes.io/projected/2fdbdc87-97f2-43ca-b3d6-099653368751-kube-api-access-rbpfv\") pod \"maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc\" (UID: \"2fdbdc87-97f2-43ca-b3d6-099653368751\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" Sep 10 22:24:59.033856 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:59.033828 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fdbdc87-97f2-43ca-b3d6-099653368751-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc\" (UID: \"2fdbdc87-97f2-43ca-b3d6-099653368751\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" Sep 10 22:24:59.033891 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:59.033874 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fdbdc87-97f2-43ca-b3d6-099653368751-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc\" (UID: \"2fdbdc87-97f2-43ca-b3d6-099653368751\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" Sep 10 22:24:59.134600 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:59.134570 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbpfv\" (UniqueName: \"kubernetes.io/projected/2fdbdc87-97f2-43ca-b3d6-099653368751-kube-api-access-rbpfv\") pod \"maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc\" (UID: \"2fdbdc87-97f2-43ca-b3d6-099653368751\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" Sep 10 22:24:59.134741 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:59.134654 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fdbdc87-97f2-43ca-b3d6-099653368751-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc\" (UID: \"2fdbdc87-97f2-43ca-b3d6-099653368751\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" Sep 10 22:24:59.134741 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:59.134699 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fdbdc87-97f2-43ca-b3d6-099653368751-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc\" (UID: \"2fdbdc87-97f2-43ca-b3d6-099653368751\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" Sep 10 22:24:59.134854 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:24:59.134786 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-serving-cert: secret "maas-api-e2e-shared-b-w2-32343f-serving-cert" not found Sep 10 22:24:59.134854 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:59.134809 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fdbdc87-97f2-43ca-b3d6-099653368751-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc\" (UID: \"2fdbdc87-97f2-43ca-b3d6-099653368751\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" Sep 10 22:24:59.135003 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:24:59.134862 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fdbdc87-97f2-43ca-b3d6-099653368751-maas-api-tls podName:2fdbdc87-97f2-43ca-b3d6-099653368751 nodeName:}" failed. No retries permitted until 2026-09-10 22:24:59.634846625 +0000 UTC m=+1201.204336572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2fdbdc87-97f2-43ca-b3d6-099653368751-maas-api-tls") pod "maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" (UID: "2fdbdc87-97f2-43ca-b3d6-099653368751") : secret "maas-api-e2e-shared-b-w2-32343f-serving-cert" not found Sep 10 22:24:59.135473 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:59.135453 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fdbdc87-97f2-43ca-b3d6-099653368751-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc\" (UID: \"2fdbdc87-97f2-43ca-b3d6-099653368751\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" Sep 10 22:24:59.137048 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:59.137028 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fdbdc87-97f2-43ca-b3d6-099653368751-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc\" (UID: \"2fdbdc87-97f2-43ca-b3d6-099653368751\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" Sep 10 22:24:59.145922 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:59.145844 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbpfv\" (UniqueName: \"kubernetes.io/projected/2fdbdc87-97f2-43ca-b3d6-099653368751-kube-api-access-rbpfv\") pod \"maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc\" (UID: \"2fdbdc87-97f2-43ca-b3d6-099653368751\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" Sep 10 22:24:59.640279 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:59.640248 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fdbdc87-97f2-43ca-b3d6-099653368751-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc\" (UID: \"2fdbdc87-97f2-43ca-b3d6-099653368751\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" Sep 10 22:24:59.642716 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:59.642693 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fdbdc87-97f2-43ca-b3d6-099653368751-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc\" (UID: \"2fdbdc87-97f2-43ca-b3d6-099653368751\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" Sep 10 22:24:59.818151 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:24:59.818122 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" Sep 10 22:25:00.091407 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:00.091383 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc"] Sep 10 22:25:00.093401 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:25:00.093373 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fdbdc87_97f2_43ca_b3d6_099653368751.slice/crio-59e86da4b34f9a797fba5fbb7874fcfd27946ea2c3c83ae2f20b5b45dd228b63 WatchSource:0}: Error finding container 59e86da4b34f9a797fba5fbb7874fcfd27946ea2c3c83ae2f20b5b45dd228b63: Status 404 returned error can't find the container with id 59e86da4b34f9a797fba5fbb7874fcfd27946ea2c3c83ae2f20b5b45dd228b63 Sep 10 22:25:00.812552 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:00.812461 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" event={"ID":"2fdbdc87-97f2-43ca-b3d6-099653368751","Type":"ContainerStarted","Data":"c70e33d8087c3bdf3cfb526eb05b09aceeeea9630f57034e95ce70feff2c88e3"} Sep 10 22:25:00.812552 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:00.812530 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" Sep 10 22:25:00.812552 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:00.812552 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" event={"ID":"2fdbdc87-97f2-43ca-b3d6-099653368751","Type":"ContainerStarted","Data":"59e86da4b34f9a797fba5fbb7874fcfd27946ea2c3c83ae2f20b5b45dd228b63"} Sep 10 22:25:00.829289 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:00.829243 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" podStartSLOduration=2.4663019840000002 podStartE2EDuration="2.829226855s" totalImagesPullingTime="362.924871ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:24:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:25:00.827822817 +0000 UTC m=+1202.397312780" watchObservedRunningTime="2026-09-10 22:25:00.829226855 +0000 UTC m=+1202.398716819" Sep 10 22:25:06.823235 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:06.823206 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" Sep 10 22:25:36.143434 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:36.143396 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf"] Sep 10 22:25:36.147269 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:36.147246 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" Sep 10 22:25:36.152231 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:36.150218 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-853c9ae4-serving-cert\"" Sep 10 22:25:36.159312 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:36.159277 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf"] Sep 10 22:25:36.265070 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:36.264954 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e164916-3706-4645-b8a3-a6e0afc6671b-openshift-service-ca\") pod \"maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf\" (UID: \"2e164916-3706-4645-b8a3-a6e0afc6671b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" Sep 10 22:25:36.265070 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:36.265010 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fqlt\" (UniqueName: \"kubernetes.io/projected/2e164916-3706-4645-b8a3-a6e0afc6671b-kube-api-access-2fqlt\") pod \"maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf\" (UID: \"2e164916-3706-4645-b8a3-a6e0afc6671b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" Sep 10 22:25:36.270160 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:36.265106 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e164916-3706-4645-b8a3-a6e0afc6671b-metrics-certs\") pod \"maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf\" (UID: \"2e164916-3706-4645-b8a3-a6e0afc6671b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" Sep 10 22:25:36.270160 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:36.265167 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e164916-3706-4645-b8a3-a6e0afc6671b-maas-api-tls\") pod \"maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf\" (UID: \"2e164916-3706-4645-b8a3-a6e0afc6671b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" Sep 10 22:25:36.366285 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:36.366253 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e164916-3706-4645-b8a3-a6e0afc6671b-maas-api-tls\") pod \"maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf\" (UID: \"2e164916-3706-4645-b8a3-a6e0afc6671b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" Sep 10 22:25:36.366464 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:25:36.366407 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-serving-cert: secret "maas-api-e2e-mt-853c9ae4-serving-cert" not found Sep 10 22:25:36.366464 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:36.366419 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e164916-3706-4645-b8a3-a6e0afc6671b-openshift-service-ca\") pod \"maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf\" (UID: \"2e164916-3706-4645-b8a3-a6e0afc6671b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" Sep 10 22:25:36.366464 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:36.366463 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fqlt\" (UniqueName: \"kubernetes.io/projected/2e164916-3706-4645-b8a3-a6e0afc6671b-kube-api-access-2fqlt\") pod \"maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf\" (UID: \"2e164916-3706-4645-b8a3-a6e0afc6671b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" Sep 10 22:25:36.366600 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:25:36.366483 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e164916-3706-4645-b8a3-a6e0afc6671b-maas-api-tls podName:2e164916-3706-4645-b8a3-a6e0afc6671b nodeName:}" failed. No retries permitted until 2026-09-10 22:25:36.866463219 +0000 UTC m=+1238.435953171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2e164916-3706-4645-b8a3-a6e0afc6671b-maas-api-tls") pod "maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" (UID: "2e164916-3706-4645-b8a3-a6e0afc6671b") : secret "maas-api-e2e-mt-853c9ae4-serving-cert" not found Sep 10 22:25:36.366600 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:36.366521 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e164916-3706-4645-b8a3-a6e0afc6671b-metrics-certs\") pod \"maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf\" (UID: \"2e164916-3706-4645-b8a3-a6e0afc6671b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" Sep 10 22:25:36.367132 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:36.367107 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e164916-3706-4645-b8a3-a6e0afc6671b-openshift-service-ca\") pod \"maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf\" (UID: \"2e164916-3706-4645-b8a3-a6e0afc6671b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" Sep 10 22:25:36.369090 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:36.369070 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e164916-3706-4645-b8a3-a6e0afc6671b-metrics-certs\") pod \"maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf\" (UID: \"2e164916-3706-4645-b8a3-a6e0afc6671b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" Sep 10 22:25:36.375499 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:36.375473 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fqlt\" (UniqueName: \"kubernetes.io/projected/2e164916-3706-4645-b8a3-a6e0afc6671b-kube-api-access-2fqlt\") pod \"maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf\" (UID: \"2e164916-3706-4645-b8a3-a6e0afc6671b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" Sep 10 22:25:36.873005 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:36.872973 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e164916-3706-4645-b8a3-a6e0afc6671b-maas-api-tls\") pod \"maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf\" (UID: \"2e164916-3706-4645-b8a3-a6e0afc6671b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" Sep 10 22:25:36.876189 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:36.876157 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e164916-3706-4645-b8a3-a6e0afc6671b-maas-api-tls\") pod \"maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf\" (UID: \"2e164916-3706-4645-b8a3-a6e0afc6671b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" Sep 10 22:25:37.062733 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:37.062699 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" Sep 10 22:25:37.337799 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:37.337771 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf"] Sep 10 22:25:37.339790 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:25:37.339764 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e164916_3706_4645_b8a3_a6e0afc6671b.slice/crio-66b37736ffcff1c49a18762048a7c6827a0f831dfdfa667a680931f5a9253230 WatchSource:0}: Error finding container 66b37736ffcff1c49a18762048a7c6827a0f831dfdfa667a680931f5a9253230: Status 404 returned error can't find the container with id 66b37736ffcff1c49a18762048a7c6827a0f831dfdfa667a680931f5a9253230 Sep 10 22:25:37.999019 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:37.998983 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" event={"ID":"2e164916-3706-4645-b8a3-a6e0afc6671b","Type":"ContainerStarted","Data":"5af1bd09f2bf47bb7a4cf6d7bd85b5fe9982aca436c9ecc373b2600afe90d70a"} Sep 10 22:25:37.999200 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:37.999034 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" event={"ID":"2e164916-3706-4645-b8a3-a6e0afc6671b","Type":"ContainerStarted","Data":"66b37736ffcff1c49a18762048a7c6827a0f831dfdfa667a680931f5a9253230"} Sep 10 22:25:37.999200 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:37.999137 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" Sep 10 22:25:38.017408 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:38.017359 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" podStartSLOduration=1.572710364 podStartE2EDuration="2.017342688s" totalImagesPullingTime="444.632324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:25:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:25:38.016254198 +0000 UTC m=+1239.585744161" watchObservedRunningTime="2026-09-10 22:25:38.017342688 +0000 UTC m=+1239.586832652" Sep 10 22:25:44.008959 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:25:44.008929 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" Sep 10 22:26:44.440632 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:26:44.440597 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf"] Sep 10 22:26:44.441200 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:26:44.440840 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" podUID="2e164916-3706-4645-b8a3-a6e0afc6671b" containerName="maas-api" containerID="cri-o://5af1bd09f2bf47bb7a4cf6d7bd85b5fe9982aca436c9ecc373b2600afe90d70a" gracePeriod=30 Sep 10 22:26:44.690513 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:26:44.690486 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" Sep 10 22:26:44.719401 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:26:44.719344 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e164916-3706-4645-b8a3-a6e0afc6671b-maas-api-tls\") pod \"2e164916-3706-4645-b8a3-a6e0afc6671b\" (UID: \"2e164916-3706-4645-b8a3-a6e0afc6671b\") " Sep 10 22:26:44.719401 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:26:44.719385 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e164916-3706-4645-b8a3-a6e0afc6671b-openshift-service-ca\") pod \"2e164916-3706-4645-b8a3-a6e0afc6671b\" (UID: \"2e164916-3706-4645-b8a3-a6e0afc6671b\") " Sep 10 22:26:44.719551 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:26:44.719433 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e164916-3706-4645-b8a3-a6e0afc6671b-metrics-certs\") pod \"2e164916-3706-4645-b8a3-a6e0afc6671b\" (UID: \"2e164916-3706-4645-b8a3-a6e0afc6671b\") " Sep 10 22:26:44.719551 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:26:44.719456 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fqlt\" (UniqueName: \"kubernetes.io/projected/2e164916-3706-4645-b8a3-a6e0afc6671b-kube-api-access-2fqlt\") pod \"2e164916-3706-4645-b8a3-a6e0afc6671b\" (UID: \"2e164916-3706-4645-b8a3-a6e0afc6671b\") " Sep 10 22:26:44.719917 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:26:44.719876 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e164916-3706-4645-b8a3-a6e0afc6671b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2e164916-3706-4645-b8a3-a6e0afc6671b" (UID: "2e164916-3706-4645-b8a3-a6e0afc6671b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:26:44.721534 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:26:44.721504 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e164916-3706-4645-b8a3-a6e0afc6671b-kube-api-access-2fqlt" (OuterVolumeSpecName: "kube-api-access-2fqlt") pod "2e164916-3706-4645-b8a3-a6e0afc6671b" (UID: "2e164916-3706-4645-b8a3-a6e0afc6671b"). InnerVolumeSpecName "kube-api-access-2fqlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:26:44.721880 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:26:44.721856 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e164916-3706-4645-b8a3-a6e0afc6671b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2e164916-3706-4645-b8a3-a6e0afc6671b" (UID: "2e164916-3706-4645-b8a3-a6e0afc6671b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:26:44.722016 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:26:44.721873 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e164916-3706-4645-b8a3-a6e0afc6671b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2e164916-3706-4645-b8a3-a6e0afc6671b" (UID: "2e164916-3706-4645-b8a3-a6e0afc6671b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:26:44.821226 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:26:44.821198 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e164916-3706-4645-b8a3-a6e0afc6671b-metrics-certs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:26:44.821226 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:26:44.821220 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fqlt\" (UniqueName: \"kubernetes.io/projected/2e164916-3706-4645-b8a3-a6e0afc6671b-kube-api-access-2fqlt\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:26:44.821226 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:26:44.821231 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e164916-3706-4645-b8a3-a6e0afc6671b-maas-api-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:26:44.821430 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:26:44.821240 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e164916-3706-4645-b8a3-a6e0afc6671b-openshift-service-ca\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:26:45.345631 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:26:45.345598 2564 generic.go:356] "Generic (PLEG): container finished" podID="2e164916-3706-4645-b8a3-a6e0afc6671b" containerID="5af1bd09f2bf47bb7a4cf6d7bd85b5fe9982aca436c9ecc373b2600afe90d70a" exitCode=0 Sep 10 22:26:45.345824 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:26:45.345643 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" Sep 10 22:26:45.345824 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:26:45.345685 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" event={"ID":"2e164916-3706-4645-b8a3-a6e0afc6671b","Type":"ContainerDied","Data":"5af1bd09f2bf47bb7a4cf6d7bd85b5fe9982aca436c9ecc373b2600afe90d70a"} Sep 10 22:26:45.345824 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:26:45.345742 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf" event={"ID":"2e164916-3706-4645-b8a3-a6e0afc6671b","Type":"ContainerDied","Data":"66b37736ffcff1c49a18762048a7c6827a0f831dfdfa667a680931f5a9253230"} Sep 10 22:26:45.345824 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:26:45.345769 2564 scope.go:117] "RemoveContainer" containerID="5af1bd09f2bf47bb7a4cf6d7bd85b5fe9982aca436c9ecc373b2600afe90d70a" Sep 10 22:26:45.355784 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:26:45.355766 2564 scope.go:117] "RemoveContainer" containerID="5af1bd09f2bf47bb7a4cf6d7bd85b5fe9982aca436c9ecc373b2600afe90d70a" Sep 10 22:26:45.356140 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:26:45.356120 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af1bd09f2bf47bb7a4cf6d7bd85b5fe9982aca436c9ecc373b2600afe90d70a\": container with ID starting with 5af1bd09f2bf47bb7a4cf6d7bd85b5fe9982aca436c9ecc373b2600afe90d70a not found: ID does not exist" containerID="5af1bd09f2bf47bb7a4cf6d7bd85b5fe9982aca436c9ecc373b2600afe90d70a" Sep 10 22:26:45.356226 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:26:45.356150 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af1bd09f2bf47bb7a4cf6d7bd85b5fe9982aca436c9ecc373b2600afe90d70a"} err="failed to get container status \"5af1bd09f2bf47bb7a4cf6d7bd85b5fe9982aca436c9ecc373b2600afe90d70a\": rpc error: code = NotFound desc = could not find container \"5af1bd09f2bf47bb7a4cf6d7bd85b5fe9982aca436c9ecc373b2600afe90d70a\": container with ID starting with 5af1bd09f2bf47bb7a4cf6d7bd85b5fe9982aca436c9ecc373b2600afe90d70a not found: ID does not exist" Sep 10 22:26:45.364755 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:26:45.364702 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf"] Sep 10 22:26:45.366375 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:26:45.366355 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-853c9ae4-6fbd769c99-9hdnf"] Sep 10 22:26:46.937455 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:26:46.937421 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e164916-3706-4645-b8a3-a6e0afc6671b" path="/var/lib/kubelet/pods/2e164916-3706-4645-b8a3-a6e0afc6671b/volumes" Sep 10 22:27:51.056153 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:51.056119 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7608-f99786"] Sep 10 22:27:51.056584 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:51.056543 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e164916-3706-4645-b8a3-a6e0afc6671b" containerName="maas-api" Sep 10 22:27:51.056584 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:51.056554 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e164916-3706-4645-b8a3-a6e0afc6671b" containerName="maas-api" Sep 10 22:27:51.056654 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:51.056616 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e164916-3706-4645-b8a3-a6e0afc6671b" containerName="maas-api" Sep 10 22:27:51.059872 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:51.059856 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-f99786" Sep 10 22:27:51.063373 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:51.063354 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qckd6\"" Sep 10 22:27:51.078098 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:51.078060 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7608-f99786"] Sep 10 22:27:51.155800 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:51.155758 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t66zq\" (UniqueName: \"kubernetes.io/projected/5a16fc9c-dd57-49c4-a24f-be9173b2fc04-kube-api-access-t66zq\") pod \"test-curl-7608-f99786\" (UID: \"5a16fc9c-dd57-49c4-a24f-be9173b2fc04\") " pod="opendatahub/test-curl-7608-f99786" Sep 10 22:27:51.256936 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:51.256883 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t66zq\" (UniqueName: \"kubernetes.io/projected/5a16fc9c-dd57-49c4-a24f-be9173b2fc04-kube-api-access-t66zq\") pod \"test-curl-7608-f99786\" (UID: \"5a16fc9c-dd57-49c4-a24f-be9173b2fc04\") " pod="opendatahub/test-curl-7608-f99786" Sep 10 22:27:51.265324 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:51.265296 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t66zq\" (UniqueName: \"kubernetes.io/projected/5a16fc9c-dd57-49c4-a24f-be9173b2fc04-kube-api-access-t66zq\") pod \"test-curl-7608-f99786\" (UID: \"5a16fc9c-dd57-49c4-a24f-be9173b2fc04\") " pod="opendatahub/test-curl-7608-f99786" Sep 10 22:27:51.368655 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:51.368620 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-f99786" Sep 10 22:27:51.638518 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:51.638493 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7608-f99786"] Sep 10 22:27:51.640492 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:27:51.640466 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a16fc9c_dd57_49c4_a24f_be9173b2fc04.slice/crio-f76c62ddf0ee4d910b17d461e51fbfeee9a255a55a0e50dc0bd84761a8213a75 WatchSource:0}: Error finding container f76c62ddf0ee4d910b17d461e51fbfeee9a255a55a0e50dc0bd84761a8213a75: Status 404 returned error can't find the container with id f76c62ddf0ee4d910b17d461e51fbfeee9a255a55a0e50dc0bd84761a8213a75 Sep 10 22:27:51.642489 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:51.642474 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 22:27:51.688950 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:51.688890 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7608-f99786" event={"ID":"5a16fc9c-dd57-49c4-a24f-be9173b2fc04","Type":"ContainerStarted","Data":"f76c62ddf0ee4d910b17d461e51fbfeee9a255a55a0e50dc0bd84761a8213a75"} Sep 10 22:27:53.701941 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:53.701876 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7608-f99786" event={"ID":"5a16fc9c-dd57-49c4-a24f-be9173b2fc04","Type":"ContainerStarted","Data":"882a99f5a02bb60579d37d38474a4b532a5611b3cdab0dd8f57a2c10d14fb862"} Sep 10 22:27:53.715704 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:53.715650 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7608-f99786" podStartSLOduration=0.875079273 podStartE2EDuration="2.715633859s" totalImagesPullingTime="1.840554586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:27:53.715103701 +0000 UTC m=+1375.284593666" watchObservedRunningTime="2026-09-10 22:27:53.715633859 +0000 UTC m=+1375.285123825" Sep 10 22:27:54.291580 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:54.291488 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7608-f99786"] Sep 10 22:27:54.295009 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:54.294981 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7608-f99786"] Sep 10 22:27:54.708484 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:27:54.708452 2564 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7608-f99786" container="test-curl-7608-f99786" Sep 10 22:27:55.715982 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:55.715941 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7608-f99786" podUID="5a16fc9c-dd57-49c4-a24f-be9173b2fc04" containerName="test-curl-7608-f99786" containerID="cri-o://882a99f5a02bb60579d37d38474a4b532a5611b3cdab0dd8f57a2c10d14fb862" gracePeriod=2 Sep 10 22:27:57.896971 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:57.896943 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-f99786" Sep 10 22:27:57.922113 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:57.922086 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t66zq\" (UniqueName: \"kubernetes.io/projected/5a16fc9c-dd57-49c4-a24f-be9173b2fc04-kube-api-access-t66zq\") pod \"5a16fc9c-dd57-49c4-a24f-be9173b2fc04\" (UID: \"5a16fc9c-dd57-49c4-a24f-be9173b2fc04\") " Sep 10 22:27:57.923991 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:57.923971 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a16fc9c-dd57-49c4-a24f-be9173b2fc04-kube-api-access-t66zq" (OuterVolumeSpecName: "kube-api-access-t66zq") pod "5a16fc9c-dd57-49c4-a24f-be9173b2fc04" (UID: "5a16fc9c-dd57-49c4-a24f-be9173b2fc04"). InnerVolumeSpecName "kube-api-access-t66zq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:27:58.022729 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:58.022640 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t66zq\" (UniqueName: \"kubernetes.io/projected/5a16fc9c-dd57-49c4-a24f-be9173b2fc04-kube-api-access-t66zq\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:27:58.733355 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:58.733321 2564 generic.go:356] "Generic (PLEG): container finished" podID="5a16fc9c-dd57-49c4-a24f-be9173b2fc04" containerID="882a99f5a02bb60579d37d38474a4b532a5611b3cdab0dd8f57a2c10d14fb862" exitCode=137 Sep 10 22:27:58.733541 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:58.733361 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-f99786" Sep 10 22:27:58.733541 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:58.733369 2564 scope.go:117] "RemoveContainer" containerID="882a99f5a02bb60579d37d38474a4b532a5611b3cdab0dd8f57a2c10d14fb862" Sep 10 22:27:58.742278 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:58.742261 2564 scope.go:117] "RemoveContainer" containerID="882a99f5a02bb60579d37d38474a4b532a5611b3cdab0dd8f57a2c10d14fb862" Sep 10 22:27:58.742534 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:27:58.742516 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"882a99f5a02bb60579d37d38474a4b532a5611b3cdab0dd8f57a2c10d14fb862\": container with ID starting with 882a99f5a02bb60579d37d38474a4b532a5611b3cdab0dd8f57a2c10d14fb862 not found: ID does not exist" containerID="882a99f5a02bb60579d37d38474a4b532a5611b3cdab0dd8f57a2c10d14fb862" Sep 10 22:27:58.742586 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:58.742542 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"882a99f5a02bb60579d37d38474a4b532a5611b3cdab0dd8f57a2c10d14fb862"} err="failed to get container status \"882a99f5a02bb60579d37d38474a4b532a5611b3cdab0dd8f57a2c10d14fb862\": rpc error: code = NotFound desc = could not find container \"882a99f5a02bb60579d37d38474a4b532a5611b3cdab0dd8f57a2c10d14fb862\": container with ID starting with 882a99f5a02bb60579d37d38474a4b532a5611b3cdab0dd8f57a2c10d14fb862 not found: ID does not exist" Sep 10 22:27:58.743255 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:27:58.743235 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7608-f99786\" is forbidden: User \"system:node:ip-10-0-137-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-148.ec2.internal' and this object" podUID="5a16fc9c-dd57-49c4-a24f-be9173b2fc04" pod="opendatahub/test-curl-7608-f99786" Sep 10 22:27:58.937057 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:27:58.937019 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7608-f99786\" is forbidden: User \"system:node:ip-10-0-137-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-148.ec2.internal' and this object" podUID="5a16fc9c-dd57-49c4-a24f-be9173b2fc04" pod="opendatahub/test-curl-7608-f99786" Sep 10 22:27:58.938429 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:27:58.938408 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a16fc9c-dd57-49c4-a24f-be9173b2fc04" path="/var/lib/kubelet/pods/5a16fc9c-dd57-49c4-a24f-be9173b2fc04/volumes" Sep 10 22:28:39.926778 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:39.926744 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6"] Sep 10 22:28:39.927243 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:39.927197 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a16fc9c-dd57-49c4-a24f-be9173b2fc04" containerName="test-curl-7608-f99786" Sep 10 22:28:39.927243 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:39.927211 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a16fc9c-dd57-49c4-a24f-be9173b2fc04" containerName="test-curl-7608-f99786" Sep 10 22:28:39.927325 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:39.927295 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a16fc9c-dd57-49c4-a24f-be9173b2fc04" containerName="test-curl-7608-f99786" Sep 10 22:28:39.930518 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:39.930502 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6" Sep 10 22:28:39.934036 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:39.934012 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-19922b-serving-cert\"" Sep 10 22:28:39.949698 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:39.949677 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6"] Sep 10 22:28:40.024617 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:40.024585 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9a64c50-63e9-4a98-a492-c1e77716838e-maas-api-tls\") pod \"maas-api-e2e-api-a-19922b-6d66b4f879-gcww6\" (UID: \"c9a64c50-63e9-4a98-a492-c1e77716838e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6" Sep 10 22:28:40.024817 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:40.024714 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9a64c50-63e9-4a98-a492-c1e77716838e-openshift-service-ca\") pod \"maas-api-e2e-api-a-19922b-6d66b4f879-gcww6\" (UID: \"c9a64c50-63e9-4a98-a492-c1e77716838e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6" Sep 10 22:28:40.024817 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:40.024755 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkj26\" (UniqueName: \"kubernetes.io/projected/c9a64c50-63e9-4a98-a492-c1e77716838e-kube-api-access-qkj26\") pod \"maas-api-e2e-api-a-19922b-6d66b4f879-gcww6\" (UID: \"c9a64c50-63e9-4a98-a492-c1e77716838e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6" Sep 10 22:28:40.024817 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:40.024774 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a64c50-63e9-4a98-a492-c1e77716838e-metrics-certs\") pod \"maas-api-e2e-api-a-19922b-6d66b4f879-gcww6\" (UID: \"c9a64c50-63e9-4a98-a492-c1e77716838e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6" Sep 10 22:28:40.125599 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:40.125569 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9a64c50-63e9-4a98-a492-c1e77716838e-openshift-service-ca\") pod \"maas-api-e2e-api-a-19922b-6d66b4f879-gcww6\" (UID: \"c9a64c50-63e9-4a98-a492-c1e77716838e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6" Sep 10 22:28:40.125799 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:40.125616 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkj26\" (UniqueName: \"kubernetes.io/projected/c9a64c50-63e9-4a98-a492-c1e77716838e-kube-api-access-qkj26\") pod \"maas-api-e2e-api-a-19922b-6d66b4f879-gcww6\" (UID: \"c9a64c50-63e9-4a98-a492-c1e77716838e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6" Sep 10 22:28:40.125799 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:40.125641 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a64c50-63e9-4a98-a492-c1e77716838e-metrics-certs\") pod \"maas-api-e2e-api-a-19922b-6d66b4f879-gcww6\" (UID: \"c9a64c50-63e9-4a98-a492-c1e77716838e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6" Sep 10 22:28:40.125799 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:40.125715 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9a64c50-63e9-4a98-a492-c1e77716838e-maas-api-tls\") pod \"maas-api-e2e-api-a-19922b-6d66b4f879-gcww6\" (UID: \"c9a64c50-63e9-4a98-a492-c1e77716838e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6" Sep 10 22:28:40.126416 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:40.126388 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9a64c50-63e9-4a98-a492-c1e77716838e-openshift-service-ca\") pod \"maas-api-e2e-api-a-19922b-6d66b4f879-gcww6\" (UID: \"c9a64c50-63e9-4a98-a492-c1e77716838e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6" Sep 10 22:28:40.128158 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:40.128135 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a64c50-63e9-4a98-a492-c1e77716838e-metrics-certs\") pod \"maas-api-e2e-api-a-19922b-6d66b4f879-gcww6\" (UID: \"c9a64c50-63e9-4a98-a492-c1e77716838e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6" Sep 10 22:28:40.128253 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:40.128179 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9a64c50-63e9-4a98-a492-c1e77716838e-maas-api-tls\") pod \"maas-api-e2e-api-a-19922b-6d66b4f879-gcww6\" (UID: \"c9a64c50-63e9-4a98-a492-c1e77716838e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6" Sep 10 22:28:40.140793 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:40.140765 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkj26\" (UniqueName: \"kubernetes.io/projected/c9a64c50-63e9-4a98-a492-c1e77716838e-kube-api-access-qkj26\") pod \"maas-api-e2e-api-a-19922b-6d66b4f879-gcww6\" (UID: \"c9a64c50-63e9-4a98-a492-c1e77716838e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6" Sep 10 22:28:40.240567 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:40.240477 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6" Sep 10 22:28:40.522667 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:40.522638 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6"] Sep 10 22:28:40.524300 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:28:40.524265 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a64c50_63e9_4a98_a492_c1e77716838e.slice/crio-90ceb05802f704c0ba6c038cf4f4c3e948ba74afc7462103e167eb45f5c46091 WatchSource:0}: Error finding container 90ceb05802f704c0ba6c038cf4f4c3e948ba74afc7462103e167eb45f5c46091: Status 404 returned error can't find the container with id 90ceb05802f704c0ba6c038cf4f4c3e948ba74afc7462103e167eb45f5c46091 Sep 10 22:28:40.949239 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:40.949202 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6" event={"ID":"c9a64c50-63e9-4a98-a492-c1e77716838e","Type":"ContainerStarted","Data":"90ceb05802f704c0ba6c038cf4f4c3e948ba74afc7462103e167eb45f5c46091"} Sep 10 22:28:41.957746 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:41.957713 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6" event={"ID":"c9a64c50-63e9-4a98-a492-c1e77716838e","Type":"ContainerStarted","Data":"506d23c25b82519eeb2c3aa006669e2e7ace5778f31e0e6b87fe15582a113ac2"} Sep 10 22:28:41.958290 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:41.957777 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6" Sep 10 22:28:41.975538 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:41.975494 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6" podStartSLOduration=2.5783318619999998 podStartE2EDuration="2.975480207s" totalImagesPullingTime="397.148345ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:28:41.973558745 +0000 UTC m=+1423.543048708" watchObservedRunningTime="2026-09-10 22:28:41.975480207 +0000 UTC m=+1423.544970170" Sep 10 22:28:47.968566 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:28:47.968536 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6" Sep 10 22:29:09.109009 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:09.108975 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl"] Sep 10 22:29:09.117060 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:09.115110 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl" Sep 10 22:29:09.119019 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:09.118995 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ad8659-serving-cert\"" Sep 10 22:29:09.123942 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:09.123921 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl"] Sep 10 22:29:09.210401 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:09.210370 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440f4b25-0d6b-40ad-9f3d-4c3f5ca44037-maas-api-tls\") pod \"maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl\" (UID: \"440f4b25-0d6b-40ad-9f3d-4c3f5ca44037\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl" Sep 10 22:29:09.210563 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:09.210407 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/440f4b25-0d6b-40ad-9f3d-4c3f5ca44037-metrics-certs\") pod \"maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl\" (UID: \"440f4b25-0d6b-40ad-9f3d-4c3f5ca44037\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl" Sep 10 22:29:09.210563 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:09.210428 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcxw4\" (UniqueName: \"kubernetes.io/projected/440f4b25-0d6b-40ad-9f3d-4c3f5ca44037-kube-api-access-fcxw4\") pod \"maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl\" (UID: \"440f4b25-0d6b-40ad-9f3d-4c3f5ca44037\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl" Sep 10 22:29:09.210563 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:09.210549 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/440f4b25-0d6b-40ad-9f3d-4c3f5ca44037-openshift-service-ca\") pod \"maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl\" (UID: \"440f4b25-0d6b-40ad-9f3d-4c3f5ca44037\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl" Sep 10 22:29:09.311082 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:09.311051 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/440f4b25-0d6b-40ad-9f3d-4c3f5ca44037-metrics-certs\") pod \"maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl\" (UID: \"440f4b25-0d6b-40ad-9f3d-4c3f5ca44037\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl" Sep 10 22:29:09.311252 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:09.311094 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcxw4\" (UniqueName: \"kubernetes.io/projected/440f4b25-0d6b-40ad-9f3d-4c3f5ca44037-kube-api-access-fcxw4\") pod \"maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl\" (UID: \"440f4b25-0d6b-40ad-9f3d-4c3f5ca44037\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl" Sep 10 22:29:09.311252 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:09.311169 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/440f4b25-0d6b-40ad-9f3d-4c3f5ca44037-openshift-service-ca\") pod \"maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl\" (UID: \"440f4b25-0d6b-40ad-9f3d-4c3f5ca44037\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl" Sep 10 22:29:09.311252 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:09.311216 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440f4b25-0d6b-40ad-9f3d-4c3f5ca44037-maas-api-tls\") pod \"maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl\" (UID: \"440f4b25-0d6b-40ad-9f3d-4c3f5ca44037\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl" Sep 10 22:29:09.311952 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:09.311926 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/440f4b25-0d6b-40ad-9f3d-4c3f5ca44037-openshift-service-ca\") pod \"maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl\" (UID: \"440f4b25-0d6b-40ad-9f3d-4c3f5ca44037\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl" Sep 10 22:29:09.313949 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:09.313879 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/440f4b25-0d6b-40ad-9f3d-4c3f5ca44037-metrics-certs\") pod \"maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl\" (UID: \"440f4b25-0d6b-40ad-9f3d-4c3f5ca44037\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl" Sep 10 22:29:09.313949 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:09.313932 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440f4b25-0d6b-40ad-9f3d-4c3f5ca44037-maas-api-tls\") pod \"maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl\" (UID: \"440f4b25-0d6b-40ad-9f3d-4c3f5ca44037\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl" Sep 10 22:29:09.324000 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:09.323981 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcxw4\" (UniqueName: \"kubernetes.io/projected/440f4b25-0d6b-40ad-9f3d-4c3f5ca44037-kube-api-access-fcxw4\") pod \"maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl\" (UID: \"440f4b25-0d6b-40ad-9f3d-4c3f5ca44037\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl" Sep 10 22:29:09.428173 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:09.428104 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl" Sep 10 22:29:09.702033 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:09.702008 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl"] Sep 10 22:29:09.703296 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:29:09.703263 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440f4b25_0d6b_40ad_9f3d_4c3f5ca44037.slice/crio-f53e80f42522c507d00b74797dbe4aaafdc7cbde00df7ff5d42b0a3af7696bea WatchSource:0}: Error finding container f53e80f42522c507d00b74797dbe4aaafdc7cbde00df7ff5d42b0a3af7696bea: Status 404 returned error can't find the container with id f53e80f42522c507d00b74797dbe4aaafdc7cbde00df7ff5d42b0a3af7696bea Sep 10 22:29:10.099418 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:10.099392 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl" event={"ID":"440f4b25-0d6b-40ad-9f3d-4c3f5ca44037","Type":"ContainerStarted","Data":"5a1268406a773ac0f4ac8bcb87fbdf50eb4b38dcc4566d07a077eb06cdcda432"} Sep 10 22:29:10.099520 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:10.099442 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl" event={"ID":"440f4b25-0d6b-40ad-9f3d-4c3f5ca44037","Type":"ContainerStarted","Data":"f53e80f42522c507d00b74797dbe4aaafdc7cbde00df7ff5d42b0a3af7696bea"} Sep 10 22:29:10.100021 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:10.100005 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl" Sep 10 22:29:10.122444 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:10.122392 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl" podStartSLOduration=0.78754003 podStartE2EDuration="1.122372752s" totalImagesPullingTime="334.832722ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:29:10.121850388 +0000 UTC m=+1451.691340349" watchObservedRunningTime="2026-09-10 22:29:10.122372752 +0000 UTC m=+1451.691862719" Sep 10 22:29:13.880815 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:13.880783 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7608-2e4e81"] Sep 10 22:29:13.884530 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:13.884514 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-2e4e81" Sep 10 22:29:13.886990 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:13.886968 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qckd6\"" Sep 10 22:29:13.889838 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:13.889817 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7608-2e4e81"] Sep 10 22:29:14.058501 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:14.058469 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzhp6\" (UniqueName: \"kubernetes.io/projected/e3d98dd2-16b3-431e-9b80-235d5eb06c61-kube-api-access-tzhp6\") pod \"test-curl-7608-2e4e81\" (UID: \"e3d98dd2-16b3-431e-9b80-235d5eb06c61\") " pod="opendatahub/test-curl-7608-2e4e81" Sep 10 22:29:14.159810 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:14.159743 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzhp6\" (UniqueName: \"kubernetes.io/projected/e3d98dd2-16b3-431e-9b80-235d5eb06c61-kube-api-access-tzhp6\") pod \"test-curl-7608-2e4e81\" (UID: \"e3d98dd2-16b3-431e-9b80-235d5eb06c61\") " pod="opendatahub/test-curl-7608-2e4e81" Sep 10 22:29:14.167088 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:14.167064 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzhp6\" (UniqueName: \"kubernetes.io/projected/e3d98dd2-16b3-431e-9b80-235d5eb06c61-kube-api-access-tzhp6\") pod \"test-curl-7608-2e4e81\" (UID: \"e3d98dd2-16b3-431e-9b80-235d5eb06c61\") " pod="opendatahub/test-curl-7608-2e4e81" Sep 10 22:29:14.195584 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:14.195559 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-2e4e81" Sep 10 22:29:14.464768 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:14.464742 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7608-2e4e81"] Sep 10 22:29:14.472937 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:29:14.466831 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3d98dd2_16b3_431e_9b80_235d5eb06c61.slice/crio-2725177b3ef09bd88d910e6f1f38f81fce2a3bc730b5af5788adbd2d88ae482a WatchSource:0}: Error finding container 2725177b3ef09bd88d910e6f1f38f81fce2a3bc730b5af5788adbd2d88ae482a: Status 404 returned error can't find the container with id 2725177b3ef09bd88d910e6f1f38f81fce2a3bc730b5af5788adbd2d88ae482a Sep 10 22:29:15.124981 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:15.124939 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7608-2e4e81" event={"ID":"e3d98dd2-16b3-431e-9b80-235d5eb06c61","Type":"ContainerStarted","Data":"074b60d69c031bc40327f4536de8a2993d1b0a0e8dc8564eab79b2dedfbad584"} Sep 10 22:29:15.124981 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:15.124986 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7608-2e4e81" event={"ID":"e3d98dd2-16b3-431e-9b80-235d5eb06c61","Type":"ContainerStarted","Data":"2725177b3ef09bd88d910e6f1f38f81fce2a3bc730b5af5788adbd2d88ae482a"} Sep 10 22:29:15.142061 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:15.142010 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7608-2e4e81" podStartSLOduration=2.141995719 podStartE2EDuration="2.141995719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:29:15.139048808 +0000 UTC m=+1456.708538773" watchObservedRunningTime="2026-09-10 22:29:15.141995719 +0000 UTC m=+1456.711485683" Sep 10 22:29:15.721002 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:15.720969 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7608-2e4e81"] Sep 10 22:29:15.724834 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:15.724808 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7608-2e4e81"] Sep 10 22:29:15.839837 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:15.839768 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7608-adbaf7"] Sep 10 22:29:15.840233 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:15.840220 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3d98dd2-16b3-431e-9b80-235d5eb06c61" containerName="test-curl-7608-2e4e81" Sep 10 22:29:15.840291 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:15.840235 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3d98dd2-16b3-431e-9b80-235d5eb06c61" containerName="test-curl-7608-2e4e81" Sep 10 22:29:15.840327 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:15.840314 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3d98dd2-16b3-431e-9b80-235d5eb06c61" containerName="test-curl-7608-2e4e81" Sep 10 22:29:15.843438 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:15.843424 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-adbaf7" Sep 10 22:29:15.849368 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:15.849345 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7608-adbaf7"] Sep 10 22:29:15.974809 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:15.974776 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l88s5\" (UniqueName: \"kubernetes.io/projected/a6e6a121-5946-4069-9b1c-6cfbfd294ee1-kube-api-access-l88s5\") pod \"test-curl-7608-adbaf7\" (UID: \"a6e6a121-5946-4069-9b1c-6cfbfd294ee1\") " pod="opendatahub/test-curl-7608-adbaf7" Sep 10 22:29:16.075764 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:16.075733 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l88s5\" (UniqueName: \"kubernetes.io/projected/a6e6a121-5946-4069-9b1c-6cfbfd294ee1-kube-api-access-l88s5\") pod \"test-curl-7608-adbaf7\" (UID: \"a6e6a121-5946-4069-9b1c-6cfbfd294ee1\") " pod="opendatahub/test-curl-7608-adbaf7" Sep 10 22:29:16.083415 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:16.083388 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l88s5\" (UniqueName: \"kubernetes.io/projected/a6e6a121-5946-4069-9b1c-6cfbfd294ee1-kube-api-access-l88s5\") pod \"test-curl-7608-adbaf7\" (UID: \"a6e6a121-5946-4069-9b1c-6cfbfd294ee1\") " pod="opendatahub/test-curl-7608-adbaf7" Sep 10 22:29:16.130319 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:29:16.130296 2564 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7608-2e4e81" container="test-curl-7608-2e4e81" Sep 10 22:29:16.154159 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:16.154142 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-adbaf7" Sep 10 22:29:16.422164 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:16.422137 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7608-adbaf7"] Sep 10 22:29:16.424188 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:29:16.424164 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6e6a121_5946_4069_9b1c_6cfbfd294ee1.slice/crio-4d3974adf337d4f8a4afaf052908141ab794c84b6e6a3f6323f8895efd0a3d16 WatchSource:0}: Error finding container 4d3974adf337d4f8a4afaf052908141ab794c84b6e6a3f6323f8895efd0a3d16: Status 404 returned error can't find the container with id 4d3974adf337d4f8a4afaf052908141ab794c84b6e6a3f6323f8895efd0a3d16 Sep 10 22:29:17.114225 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:17.114197 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl" Sep 10 22:29:17.137188 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:17.137154 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7608-adbaf7" event={"ID":"a6e6a121-5946-4069-9b1c-6cfbfd294ee1","Type":"ContainerStarted","Data":"6b7ef4ba202c0db760705e587deecd135c9145f1d018d443c29ef80bcc81217a"} Sep 10 22:29:17.137587 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:17.137196 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7608-adbaf7" event={"ID":"a6e6a121-5946-4069-9b1c-6cfbfd294ee1","Type":"ContainerStarted","Data":"4d3974adf337d4f8a4afaf052908141ab794c84b6e6a3f6323f8895efd0a3d16"} Sep 10 22:29:17.137587 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:17.137269 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7608-2e4e81" podUID="e3d98dd2-16b3-431e-9b80-235d5eb06c61" containerName="test-curl-7608-2e4e81" containerID="cri-o://074b60d69c031bc40327f4536de8a2993d1b0a0e8dc8564eab79b2dedfbad584" gracePeriod=2 Sep 10 22:29:17.156610 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:17.156550 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7608-adbaf7" podStartSLOduration=2.156531728 podStartE2EDuration="2.156531728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:29:17.153888784 +0000 UTC m=+1458.723378748" watchObservedRunningTime="2026-09-10 22:29:17.156531728 +0000 UTC m=+1458.726021691" Sep 10 22:29:17.806324 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:17.806287 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7608-adbaf7"] Sep 10 22:29:17.808156 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:17.808132 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7608-adbaf7"] Sep 10 22:29:18.142204 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:29:18.142177 2564 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7608-adbaf7" container="test-curl-7608-adbaf7" Sep 10 22:29:19.147796 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:19.147744 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7608-adbaf7" podUID="a6e6a121-5946-4069-9b1c-6cfbfd294ee1" containerName="test-curl-7608-adbaf7" containerID="cri-o://6b7ef4ba202c0db760705e587deecd135c9145f1d018d443c29ef80bcc81217a" gracePeriod=2 Sep 10 22:29:19.326240 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:19.326214 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-2e4e81" Sep 10 22:29:19.406299 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:19.406238 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzhp6\" (UniqueName: \"kubernetes.io/projected/e3d98dd2-16b3-431e-9b80-235d5eb06c61-kube-api-access-tzhp6\") pod \"e3d98dd2-16b3-431e-9b80-235d5eb06c61\" (UID: \"e3d98dd2-16b3-431e-9b80-235d5eb06c61\") " Sep 10 22:29:19.408081 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:19.408056 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d98dd2-16b3-431e-9b80-235d5eb06c61-kube-api-access-tzhp6" (OuterVolumeSpecName: "kube-api-access-tzhp6") pod "e3d98dd2-16b3-431e-9b80-235d5eb06c61" (UID: "e3d98dd2-16b3-431e-9b80-235d5eb06c61"). InnerVolumeSpecName "kube-api-access-tzhp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:29:19.507881 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:19.507850 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzhp6\" (UniqueName: \"kubernetes.io/projected/e3d98dd2-16b3-431e-9b80-235d5eb06c61-kube-api-access-tzhp6\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:29:20.156095 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:20.156051 2564 generic.go:356] "Generic (PLEG): container finished" podID="e3d98dd2-16b3-431e-9b80-235d5eb06c61" containerID="074b60d69c031bc40327f4536de8a2993d1b0a0e8dc8564eab79b2dedfbad584" exitCode=137 Sep 10 22:29:20.156095 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:20.156097 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-2e4e81" Sep 10 22:29:20.156594 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:20.156145 2564 scope.go:117] "RemoveContainer" containerID="074b60d69c031bc40327f4536de8a2993d1b0a0e8dc8564eab79b2dedfbad584" Sep 10 22:29:20.165453 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:20.165437 2564 scope.go:117] "RemoveContainer" containerID="074b60d69c031bc40327f4536de8a2993d1b0a0e8dc8564eab79b2dedfbad584" Sep 10 22:29:20.165751 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:29:20.165732 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"074b60d69c031bc40327f4536de8a2993d1b0a0e8dc8564eab79b2dedfbad584\": container with ID starting with 074b60d69c031bc40327f4536de8a2993d1b0a0e8dc8564eab79b2dedfbad584 not found: ID does not exist" containerID="074b60d69c031bc40327f4536de8a2993d1b0a0e8dc8564eab79b2dedfbad584" Sep 10 22:29:20.165806 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:20.165759 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"074b60d69c031bc40327f4536de8a2993d1b0a0e8dc8564eab79b2dedfbad584"} err="failed to get container status \"074b60d69c031bc40327f4536de8a2993d1b0a0e8dc8564eab79b2dedfbad584\": rpc error: code = NotFound desc = could not find container \"074b60d69c031bc40327f4536de8a2993d1b0a0e8dc8564eab79b2dedfbad584\": container with ID starting with 074b60d69c031bc40327f4536de8a2993d1b0a0e8dc8564eab79b2dedfbad584 not found: ID does not exist" Sep 10 22:29:20.166974 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:29:20.166951 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7608-2e4e81\" is forbidden: User \"system:node:ip-10-0-137-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-148.ec2.internal' and this object" podUID="e3d98dd2-16b3-431e-9b80-235d5eb06c61" pod="opendatahub/test-curl-7608-2e4e81" Sep 10 22:29:20.938110 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:20.938083 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3d98dd2-16b3-431e-9b80-235d5eb06c61" path="/var/lib/kubelet/pods/e3d98dd2-16b3-431e-9b80-235d5eb06c61/volumes" Sep 10 22:29:21.344770 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:21.344748 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-adbaf7" Sep 10 22:29:21.425780 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:21.425745 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l88s5\" (UniqueName: \"kubernetes.io/projected/a6e6a121-5946-4069-9b1c-6cfbfd294ee1-kube-api-access-l88s5\") pod \"a6e6a121-5946-4069-9b1c-6cfbfd294ee1\" (UID: \"a6e6a121-5946-4069-9b1c-6cfbfd294ee1\") " Sep 10 22:29:21.427676 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:21.427646 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e6a121-5946-4069-9b1c-6cfbfd294ee1-kube-api-access-l88s5" (OuterVolumeSpecName: "kube-api-access-l88s5") pod "a6e6a121-5946-4069-9b1c-6cfbfd294ee1" (UID: "a6e6a121-5946-4069-9b1c-6cfbfd294ee1"). InnerVolumeSpecName "kube-api-access-l88s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:29:21.526839 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:21.526778 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l88s5\" (UniqueName: \"kubernetes.io/projected/a6e6a121-5946-4069-9b1c-6cfbfd294ee1-kube-api-access-l88s5\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:29:22.168205 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:22.168169 2564 generic.go:356] "Generic (PLEG): container finished" podID="a6e6a121-5946-4069-9b1c-6cfbfd294ee1" containerID="6b7ef4ba202c0db760705e587deecd135c9145f1d018d443c29ef80bcc81217a" exitCode=137 Sep 10 22:29:22.168393 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:22.168211 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-adbaf7" Sep 10 22:29:22.168393 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:22.168282 2564 scope.go:117] "RemoveContainer" containerID="6b7ef4ba202c0db760705e587deecd135c9145f1d018d443c29ef80bcc81217a" Sep 10 22:29:22.176944 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:22.176924 2564 scope.go:117] "RemoveContainer" containerID="6b7ef4ba202c0db760705e587deecd135c9145f1d018d443c29ef80bcc81217a" Sep 10 22:29:22.177198 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:29:22.177181 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b7ef4ba202c0db760705e587deecd135c9145f1d018d443c29ef80bcc81217a\": container with ID starting with 6b7ef4ba202c0db760705e587deecd135c9145f1d018d443c29ef80bcc81217a not found: ID does not exist" containerID="6b7ef4ba202c0db760705e587deecd135c9145f1d018d443c29ef80bcc81217a" Sep 10 22:29:22.177259 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:22.177206 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b7ef4ba202c0db760705e587deecd135c9145f1d018d443c29ef80bcc81217a"} err="failed to get container status \"6b7ef4ba202c0db760705e587deecd135c9145f1d018d443c29ef80bcc81217a\": rpc error: code = NotFound desc = could not find container \"6b7ef4ba202c0db760705e587deecd135c9145f1d018d443c29ef80bcc81217a\": container with ID starting with 6b7ef4ba202c0db760705e587deecd135c9145f1d018d443c29ef80bcc81217a not found: ID does not exist" Sep 10 22:29:22.177868 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:29:22.177852 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7608-adbaf7\" is forbidden: User \"system:node:ip-10-0-137-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-148.ec2.internal' and this object" podUID="a6e6a121-5946-4069-9b1c-6cfbfd294ee1" pod="opendatahub/test-curl-7608-adbaf7" Sep 10 22:29:22.939661 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:22.939620 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6e6a121-5946-4069-9b1c-6cfbfd294ee1" path="/var/lib/kubelet/pods/a6e6a121-5946-4069-9b1c-6cfbfd294ee1/volumes" Sep 10 22:29:51.867577 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:51.867543 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6"] Sep 10 22:29:51.868494 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:51.868459 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6" podUID="c9a64c50-63e9-4a98-a492-c1e77716838e" containerName="maas-api" containerID="cri-o://506d23c25b82519eeb2c3aa006669e2e7ace5778f31e0e6b87fe15582a113ac2" gracePeriod=30 Sep 10 22:29:52.114499 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:52.114473 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6" Sep 10 22:29:52.235098 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:52.235021 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkj26\" (UniqueName: \"kubernetes.io/projected/c9a64c50-63e9-4a98-a492-c1e77716838e-kube-api-access-qkj26\") pod \"c9a64c50-63e9-4a98-a492-c1e77716838e\" (UID: \"c9a64c50-63e9-4a98-a492-c1e77716838e\") " Sep 10 22:29:52.235242 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:52.235200 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9a64c50-63e9-4a98-a492-c1e77716838e-maas-api-tls\") pod \"c9a64c50-63e9-4a98-a492-c1e77716838e\" (UID: \"c9a64c50-63e9-4a98-a492-c1e77716838e\") " Sep 10 22:29:52.235360 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:52.235263 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a64c50-63e9-4a98-a492-c1e77716838e-metrics-certs\") pod \"c9a64c50-63e9-4a98-a492-c1e77716838e\" (UID: \"c9a64c50-63e9-4a98-a492-c1e77716838e\") " Sep 10 22:29:52.235424 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:52.235374 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9a64c50-63e9-4a98-a492-c1e77716838e-openshift-service-ca\") pod \"c9a64c50-63e9-4a98-a492-c1e77716838e\" (UID: \"c9a64c50-63e9-4a98-a492-c1e77716838e\") " Sep 10 22:29:52.235788 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:52.235755 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9a64c50-63e9-4a98-a492-c1e77716838e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c9a64c50-63e9-4a98-a492-c1e77716838e" (UID: "c9a64c50-63e9-4a98-a492-c1e77716838e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:29:52.237280 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:52.237256 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9a64c50-63e9-4a98-a492-c1e77716838e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c9a64c50-63e9-4a98-a492-c1e77716838e" (UID: "c9a64c50-63e9-4a98-a492-c1e77716838e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:29:52.237577 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:52.237557 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a64c50-63e9-4a98-a492-c1e77716838e-kube-api-access-qkj26" (OuterVolumeSpecName: "kube-api-access-qkj26") pod "c9a64c50-63e9-4a98-a492-c1e77716838e" (UID: "c9a64c50-63e9-4a98-a492-c1e77716838e"). InnerVolumeSpecName "kube-api-access-qkj26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:29:52.237655 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:52.237559 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9a64c50-63e9-4a98-a492-c1e77716838e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c9a64c50-63e9-4a98-a492-c1e77716838e" (UID: "c9a64c50-63e9-4a98-a492-c1e77716838e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:29:52.329603 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:52.329561 2564 generic.go:356] "Generic (PLEG): container finished" podID="c9a64c50-63e9-4a98-a492-c1e77716838e" containerID="506d23c25b82519eeb2c3aa006669e2e7ace5778f31e0e6b87fe15582a113ac2" exitCode=0 Sep 10 22:29:52.329603 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:52.329608 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6" Sep 10 22:29:52.329866 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:52.329634 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6" event={"ID":"c9a64c50-63e9-4a98-a492-c1e77716838e","Type":"ContainerDied","Data":"506d23c25b82519eeb2c3aa006669e2e7ace5778f31e0e6b87fe15582a113ac2"} Sep 10 22:29:52.329866 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:52.329690 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6" event={"ID":"c9a64c50-63e9-4a98-a492-c1e77716838e","Type":"ContainerDied","Data":"90ceb05802f704c0ba6c038cf4f4c3e948ba74afc7462103e167eb45f5c46091"} Sep 10 22:29:52.329866 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:52.329714 2564 scope.go:117] "RemoveContainer" containerID="506d23c25b82519eeb2c3aa006669e2e7ace5778f31e0e6b87fe15582a113ac2" Sep 10 22:29:52.336726 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:52.336648 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a64c50-63e9-4a98-a492-c1e77716838e-metrics-certs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:29:52.336726 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:52.336677 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9a64c50-63e9-4a98-a492-c1e77716838e-openshift-service-ca\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:29:52.336726 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:52.336690 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkj26\" (UniqueName: \"kubernetes.io/projected/c9a64c50-63e9-4a98-a492-c1e77716838e-kube-api-access-qkj26\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:29:52.336726 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:52.336705 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9a64c50-63e9-4a98-a492-c1e77716838e-maas-api-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:29:52.340603 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:52.340581 2564 scope.go:117] "RemoveContainer" containerID="506d23c25b82519eeb2c3aa006669e2e7ace5778f31e0e6b87fe15582a113ac2" Sep 10 22:29:52.340948 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:29:52.340928 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"506d23c25b82519eeb2c3aa006669e2e7ace5778f31e0e6b87fe15582a113ac2\": container with ID starting with 506d23c25b82519eeb2c3aa006669e2e7ace5778f31e0e6b87fe15582a113ac2 not found: ID does not exist" containerID="506d23c25b82519eeb2c3aa006669e2e7ace5778f31e0e6b87fe15582a113ac2" Sep 10 22:29:52.341014 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:52.340957 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"506d23c25b82519eeb2c3aa006669e2e7ace5778f31e0e6b87fe15582a113ac2"} err="failed to get container status \"506d23c25b82519eeb2c3aa006669e2e7ace5778f31e0e6b87fe15582a113ac2\": rpc error: code = NotFound desc = could not find container \"506d23c25b82519eeb2c3aa006669e2e7ace5778f31e0e6b87fe15582a113ac2\": container with ID starting with 506d23c25b82519eeb2c3aa006669e2e7ace5778f31e0e6b87fe15582a113ac2 not found: ID does not exist" Sep 10 22:29:52.352580 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:52.352555 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6"] Sep 10 22:29:52.356700 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:52.356678 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-19922b-6d66b4f879-gcww6"] Sep 10 22:29:52.937911 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:29:52.937876 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9a64c50-63e9-4a98-a492-c1e77716838e" path="/var/lib/kubelet/pods/c9a64c50-63e9-4a98-a492-c1e77716838e/volumes" Sep 10 22:30:14.182412 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:14.182380 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl"] Sep 10 22:30:14.182991 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:14.182648 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl" podUID="440f4b25-0d6b-40ad-9f3d-4c3f5ca44037" containerName="maas-api" containerID="cri-o://5a1268406a773ac0f4ac8bcb87fbdf50eb4b38dcc4566d07a077eb06cdcda432" gracePeriod=30 Sep 10 22:30:14.433420 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:14.433359 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl" Sep 10 22:30:14.447854 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:14.447819 2564 generic.go:356] "Generic (PLEG): container finished" podID="440f4b25-0d6b-40ad-9f3d-4c3f5ca44037" containerID="5a1268406a773ac0f4ac8bcb87fbdf50eb4b38dcc4566d07a077eb06cdcda432" exitCode=0 Sep 10 22:30:14.448100 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:14.447859 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl" Sep 10 22:30:14.448100 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:14.447924 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl" event={"ID":"440f4b25-0d6b-40ad-9f3d-4c3f5ca44037","Type":"ContainerDied","Data":"5a1268406a773ac0f4ac8bcb87fbdf50eb4b38dcc4566d07a077eb06cdcda432"} Sep 10 22:30:14.448100 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:14.447978 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl" event={"ID":"440f4b25-0d6b-40ad-9f3d-4c3f5ca44037","Type":"ContainerDied","Data":"f53e80f42522c507d00b74797dbe4aaafdc7cbde00df7ff5d42b0a3af7696bea"} Sep 10 22:30:14.448100 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:14.448005 2564 scope.go:117] "RemoveContainer" containerID="5a1268406a773ac0f4ac8bcb87fbdf50eb4b38dcc4566d07a077eb06cdcda432" Sep 10 22:30:14.458802 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:14.458774 2564 scope.go:117] "RemoveContainer" containerID="5a1268406a773ac0f4ac8bcb87fbdf50eb4b38dcc4566d07a077eb06cdcda432" Sep 10 22:30:14.459230 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:30:14.459207 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a1268406a773ac0f4ac8bcb87fbdf50eb4b38dcc4566d07a077eb06cdcda432\": container with ID starting with 5a1268406a773ac0f4ac8bcb87fbdf50eb4b38dcc4566d07a077eb06cdcda432 not found: ID does not exist" containerID="5a1268406a773ac0f4ac8bcb87fbdf50eb4b38dcc4566d07a077eb06cdcda432" Sep 10 22:30:14.459321 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:14.459245 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a1268406a773ac0f4ac8bcb87fbdf50eb4b38dcc4566d07a077eb06cdcda432"} err="failed to get container status \"5a1268406a773ac0f4ac8bcb87fbdf50eb4b38dcc4566d07a077eb06cdcda432\": rpc error: code = NotFound desc = could not find container \"5a1268406a773ac0f4ac8bcb87fbdf50eb4b38dcc4566d07a077eb06cdcda432\": container with ID starting with 5a1268406a773ac0f4ac8bcb87fbdf50eb4b38dcc4566d07a077eb06cdcda432 not found: ID does not exist" Sep 10 22:30:14.548791 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:14.548754 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/440f4b25-0d6b-40ad-9f3d-4c3f5ca44037-openshift-service-ca\") pod \"440f4b25-0d6b-40ad-9f3d-4c3f5ca44037\" (UID: \"440f4b25-0d6b-40ad-9f3d-4c3f5ca44037\") " Sep 10 22:30:14.548992 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:14.548834 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/440f4b25-0d6b-40ad-9f3d-4c3f5ca44037-metrics-certs\") pod \"440f4b25-0d6b-40ad-9f3d-4c3f5ca44037\" (UID: \"440f4b25-0d6b-40ad-9f3d-4c3f5ca44037\") " Sep 10 22:30:14.548992 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:14.548895 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcxw4\" (UniqueName: \"kubernetes.io/projected/440f4b25-0d6b-40ad-9f3d-4c3f5ca44037-kube-api-access-fcxw4\") pod \"440f4b25-0d6b-40ad-9f3d-4c3f5ca44037\" (UID: \"440f4b25-0d6b-40ad-9f3d-4c3f5ca44037\") " Sep 10 22:30:14.548992 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:14.548986 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440f4b25-0d6b-40ad-9f3d-4c3f5ca44037-maas-api-tls\") pod \"440f4b25-0d6b-40ad-9f3d-4c3f5ca44037\" (UID: \"440f4b25-0d6b-40ad-9f3d-4c3f5ca44037\") " Sep 10 22:30:14.549206 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:14.549183 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440f4b25-0d6b-40ad-9f3d-4c3f5ca44037-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "440f4b25-0d6b-40ad-9f3d-4c3f5ca44037" (UID: "440f4b25-0d6b-40ad-9f3d-4c3f5ca44037"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:30:14.549332 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:14.549315 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/440f4b25-0d6b-40ad-9f3d-4c3f5ca44037-openshift-service-ca\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:30:14.551110 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:14.551079 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440f4b25-0d6b-40ad-9f3d-4c3f5ca44037-kube-api-access-fcxw4" (OuterVolumeSpecName: "kube-api-access-fcxw4") pod "440f4b25-0d6b-40ad-9f3d-4c3f5ca44037" (UID: "440f4b25-0d6b-40ad-9f3d-4c3f5ca44037"). InnerVolumeSpecName "kube-api-access-fcxw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:30:14.551261 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:14.551241 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440f4b25-0d6b-40ad-9f3d-4c3f5ca44037-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "440f4b25-0d6b-40ad-9f3d-4c3f5ca44037" (UID: "440f4b25-0d6b-40ad-9f3d-4c3f5ca44037"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:30:14.551424 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:14.551409 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440f4b25-0d6b-40ad-9f3d-4c3f5ca44037-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "440f4b25-0d6b-40ad-9f3d-4c3f5ca44037" (UID: "440f4b25-0d6b-40ad-9f3d-4c3f5ca44037"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:30:14.650473 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:14.650432 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/440f4b25-0d6b-40ad-9f3d-4c3f5ca44037-metrics-certs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:30:14.650473 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:14.650469 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcxw4\" (UniqueName: \"kubernetes.io/projected/440f4b25-0d6b-40ad-9f3d-4c3f5ca44037-kube-api-access-fcxw4\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:30:14.650473 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:14.650479 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440f4b25-0d6b-40ad-9f3d-4c3f5ca44037-maas-api-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:30:14.774043 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:14.774011 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl"] Sep 10 22:30:14.778050 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:14.778018 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad8659-7c695b95b6-6wnzl"] Sep 10 22:30:14.937498 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:14.937463 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="440f4b25-0d6b-40ad-9f3d-4c3f5ca44037" path="/var/lib/kubelet/pods/440f4b25-0d6b-40ad-9f3d-4c3f5ca44037/volumes" Sep 10 22:30:31.525551 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:31.525517 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg"] Sep 10 22:30:31.526194 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:31.525759 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg" podUID="4b644ebc-739f-451b-b8c0-b2fd6852d45a" containerName="maas-api" containerID="cri-o://4d5643110ffc733b285082f4c54c7ded8ba6ac06f4a684db2721b429f8c022c2" gracePeriod=30 Sep 10 22:30:31.776476 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:31.776423 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg" Sep 10 22:30:31.912181 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:31.912152 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b644ebc-739f-451b-b8c0-b2fd6852d45a-metrics-certs\") pod \"4b644ebc-739f-451b-b8c0-b2fd6852d45a\" (UID: \"4b644ebc-739f-451b-b8c0-b2fd6852d45a\") " Sep 10 22:30:31.912346 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:31.912233 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b644ebc-739f-451b-b8c0-b2fd6852d45a-openshift-service-ca\") pod \"4b644ebc-739f-451b-b8c0-b2fd6852d45a\" (UID: \"4b644ebc-739f-451b-b8c0-b2fd6852d45a\") " Sep 10 22:30:31.912346 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:31.912291 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b644ebc-739f-451b-b8c0-b2fd6852d45a-maas-api-tls\") pod \"4b644ebc-739f-451b-b8c0-b2fd6852d45a\" (UID: \"4b644ebc-739f-451b-b8c0-b2fd6852d45a\") " Sep 10 22:30:31.912475 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:31.912394 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pchkc\" (UniqueName: \"kubernetes.io/projected/4b644ebc-739f-451b-b8c0-b2fd6852d45a-kube-api-access-pchkc\") pod \"4b644ebc-739f-451b-b8c0-b2fd6852d45a\" (UID: \"4b644ebc-739f-451b-b8c0-b2fd6852d45a\") " Sep 10 22:30:31.912534 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:31.912504 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b644ebc-739f-451b-b8c0-b2fd6852d45a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4b644ebc-739f-451b-b8c0-b2fd6852d45a" (UID: "4b644ebc-739f-451b-b8c0-b2fd6852d45a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:30:31.912742 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:31.912721 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b644ebc-739f-451b-b8c0-b2fd6852d45a-openshift-service-ca\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:30:31.914222 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:31.914195 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b644ebc-739f-451b-b8c0-b2fd6852d45a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4b644ebc-739f-451b-b8c0-b2fd6852d45a" (UID: "4b644ebc-739f-451b-b8c0-b2fd6852d45a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:30:31.914222 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:31.914206 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b644ebc-739f-451b-b8c0-b2fd6852d45a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4b644ebc-739f-451b-b8c0-b2fd6852d45a" (UID: "4b644ebc-739f-451b-b8c0-b2fd6852d45a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:30:31.914349 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:31.914317 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b644ebc-739f-451b-b8c0-b2fd6852d45a-kube-api-access-pchkc" (OuterVolumeSpecName: "kube-api-access-pchkc") pod "4b644ebc-739f-451b-b8c0-b2fd6852d45a" (UID: "4b644ebc-739f-451b-b8c0-b2fd6852d45a"). InnerVolumeSpecName "kube-api-access-pchkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:30:32.013641 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:32.013618 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b644ebc-739f-451b-b8c0-b2fd6852d45a-maas-api-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:30:32.013641 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:32.013640 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pchkc\" (UniqueName: \"kubernetes.io/projected/4b644ebc-739f-451b-b8c0-b2fd6852d45a-kube-api-access-pchkc\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:30:32.013772 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:32.013650 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b644ebc-739f-451b-b8c0-b2fd6852d45a-metrics-certs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:30:32.545591 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:32.545556 2564 generic.go:356] "Generic (PLEG): container finished" podID="4b644ebc-739f-451b-b8c0-b2fd6852d45a" containerID="4d5643110ffc733b285082f4c54c7ded8ba6ac06f4a684db2721b429f8c022c2" exitCode=0 Sep 10 22:30:32.546086 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:32.545682 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg" event={"ID":"4b644ebc-739f-451b-b8c0-b2fd6852d45a","Type":"ContainerDied","Data":"4d5643110ffc733b285082f4c54c7ded8ba6ac06f4a684db2721b429f8c022c2"} Sep 10 22:30:32.546086 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:32.545704 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg" Sep 10 22:30:32.546086 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:32.545761 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg" event={"ID":"4b644ebc-739f-451b-b8c0-b2fd6852d45a","Type":"ContainerDied","Data":"eaa0ddfe9502041d4480aaebca2a63f96b9db35db6ca4af33fb01a563e4131f0"} Sep 10 22:30:32.546086 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:32.545794 2564 scope.go:117] "RemoveContainer" containerID="4d5643110ffc733b285082f4c54c7ded8ba6ac06f4a684db2721b429f8c022c2" Sep 10 22:30:32.557050 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:32.557034 2564 scope.go:117] "RemoveContainer" containerID="4d5643110ffc733b285082f4c54c7ded8ba6ac06f4a684db2721b429f8c022c2" Sep 10 22:30:32.557314 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:30:32.557291 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d5643110ffc733b285082f4c54c7ded8ba6ac06f4a684db2721b429f8c022c2\": container with ID starting with 4d5643110ffc733b285082f4c54c7ded8ba6ac06f4a684db2721b429f8c022c2 not found: ID does not exist" containerID="4d5643110ffc733b285082f4c54c7ded8ba6ac06f4a684db2721b429f8c022c2" Sep 10 22:30:32.557399 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:32.557329 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d5643110ffc733b285082f4c54c7ded8ba6ac06f4a684db2721b429f8c022c2"} err="failed to get container status \"4d5643110ffc733b285082f4c54c7ded8ba6ac06f4a684db2721b429f8c022c2\": rpc error: code = NotFound desc = could not find container \"4d5643110ffc733b285082f4c54c7ded8ba6ac06f4a684db2721b429f8c022c2\": container with ID starting with 4d5643110ffc733b285082f4c54c7ded8ba6ac06f4a684db2721b429f8c022c2 not found: ID does not exist" Sep 10 22:30:32.570992 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:32.570970 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg"] Sep 10 22:30:32.580758 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:32.580734 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4a864c-6f79879d5-4x7fg"] Sep 10 22:30:32.943288 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:32.943254 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b644ebc-739f-451b-b8c0-b2fd6852d45a" path="/var/lib/kubelet/pods/4b644ebc-739f-451b-b8c0-b2fd6852d45a/volumes" Sep 10 22:30:48.885451 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:48.885419 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc"] Sep 10 22:30:48.887987 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:48.886143 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" podUID="2fdbdc87-97f2-43ca-b3d6-099653368751" containerName="maas-api" containerID="cri-o://c70e33d8087c3bdf3cfb526eb05b09aceeeea9630f57034e95ce70feff2c88e3" gracePeriod=30 Sep 10 22:30:49.136443 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:49.136385 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" Sep 10 22:30:49.162657 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:49.162632 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fdbdc87-97f2-43ca-b3d6-099653368751-metrics-certs\") pod \"2fdbdc87-97f2-43ca-b3d6-099653368751\" (UID: \"2fdbdc87-97f2-43ca-b3d6-099653368751\") " Sep 10 22:30:49.162802 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:49.162662 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fdbdc87-97f2-43ca-b3d6-099653368751-openshift-service-ca\") pod \"2fdbdc87-97f2-43ca-b3d6-099653368751\" (UID: \"2fdbdc87-97f2-43ca-b3d6-099653368751\") " Sep 10 22:30:49.162802 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:49.162683 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fdbdc87-97f2-43ca-b3d6-099653368751-maas-api-tls\") pod \"2fdbdc87-97f2-43ca-b3d6-099653368751\" (UID: \"2fdbdc87-97f2-43ca-b3d6-099653368751\") " Sep 10 22:30:49.162802 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:49.162699 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbpfv\" (UniqueName: \"kubernetes.io/projected/2fdbdc87-97f2-43ca-b3d6-099653368751-kube-api-access-rbpfv\") pod \"2fdbdc87-97f2-43ca-b3d6-099653368751\" (UID: \"2fdbdc87-97f2-43ca-b3d6-099653368751\") " Sep 10 22:30:49.163096 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:49.163065 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fdbdc87-97f2-43ca-b3d6-099653368751-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2fdbdc87-97f2-43ca-b3d6-099653368751" (UID: "2fdbdc87-97f2-43ca-b3d6-099653368751"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:30:49.164724 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:49.164700 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fdbdc87-97f2-43ca-b3d6-099653368751-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2fdbdc87-97f2-43ca-b3d6-099653368751" (UID: "2fdbdc87-97f2-43ca-b3d6-099653368751"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:30:49.164822 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:49.164793 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fdbdc87-97f2-43ca-b3d6-099653368751-kube-api-access-rbpfv" (OuterVolumeSpecName: "kube-api-access-rbpfv") pod "2fdbdc87-97f2-43ca-b3d6-099653368751" (UID: "2fdbdc87-97f2-43ca-b3d6-099653368751"). InnerVolumeSpecName "kube-api-access-rbpfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:30:49.165128 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:49.165111 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fdbdc87-97f2-43ca-b3d6-099653368751-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2fdbdc87-97f2-43ca-b3d6-099653368751" (UID: "2fdbdc87-97f2-43ca-b3d6-099653368751"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:30:49.263558 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:49.263528 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fdbdc87-97f2-43ca-b3d6-099653368751-metrics-certs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:30:49.263558 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:49.263551 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fdbdc87-97f2-43ca-b3d6-099653368751-openshift-service-ca\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:30:49.263558 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:49.263560 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fdbdc87-97f2-43ca-b3d6-099653368751-maas-api-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:30:49.263791 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:49.263570 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbpfv\" (UniqueName: \"kubernetes.io/projected/2fdbdc87-97f2-43ca-b3d6-099653368751-kube-api-access-rbpfv\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:30:49.633629 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:49.633597 2564 generic.go:356] "Generic (PLEG): container finished" podID="2fdbdc87-97f2-43ca-b3d6-099653368751" containerID="c70e33d8087c3bdf3cfb526eb05b09aceeeea9630f57034e95ce70feff2c88e3" exitCode=0 Sep 10 22:30:49.633822 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:49.633647 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" Sep 10 22:30:49.633822 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:49.633685 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" event={"ID":"2fdbdc87-97f2-43ca-b3d6-099653368751","Type":"ContainerDied","Data":"c70e33d8087c3bdf3cfb526eb05b09aceeeea9630f57034e95ce70feff2c88e3"} Sep 10 22:30:49.633822 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:49.633738 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc" event={"ID":"2fdbdc87-97f2-43ca-b3d6-099653368751","Type":"ContainerDied","Data":"59e86da4b34f9a797fba5fbb7874fcfd27946ea2c3c83ae2f20b5b45dd228b63"} Sep 10 22:30:49.633822 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:49.633763 2564 scope.go:117] "RemoveContainer" containerID="c70e33d8087c3bdf3cfb526eb05b09aceeeea9630f57034e95ce70feff2c88e3" Sep 10 22:30:49.642654 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:49.642641 2564 scope.go:117] "RemoveContainer" containerID="c70e33d8087c3bdf3cfb526eb05b09aceeeea9630f57034e95ce70feff2c88e3" Sep 10 22:30:49.642884 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:30:49.642866 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c70e33d8087c3bdf3cfb526eb05b09aceeeea9630f57034e95ce70feff2c88e3\": container with ID starting with c70e33d8087c3bdf3cfb526eb05b09aceeeea9630f57034e95ce70feff2c88e3 not found: ID does not exist" containerID="c70e33d8087c3bdf3cfb526eb05b09aceeeea9630f57034e95ce70feff2c88e3" Sep 10 22:30:49.642954 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:49.642892 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c70e33d8087c3bdf3cfb526eb05b09aceeeea9630f57034e95ce70feff2c88e3"} err="failed to get container status \"c70e33d8087c3bdf3cfb526eb05b09aceeeea9630f57034e95ce70feff2c88e3\": rpc error: code = NotFound desc = could not find container \"c70e33d8087c3bdf3cfb526eb05b09aceeeea9630f57034e95ce70feff2c88e3\": container with ID starting with c70e33d8087c3bdf3cfb526eb05b09aceeeea9630f57034e95ce70feff2c88e3 not found: ID does not exist" Sep 10 22:30:49.671515 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:49.671492 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc"] Sep 10 22:30:49.684953 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:49.684932 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-32343f-5c59446dc5-f6pvc"] Sep 10 22:30:50.937609 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:30:50.937572 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fdbdc87-97f2-43ca-b3d6-099653368751" path="/var/lib/kubelet/pods/2fdbdc87-97f2-43ca-b3d6-099653368751/volumes" Sep 10 22:31:17.252422 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.251688 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr"] Sep 10 22:31:17.253040 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.252975 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b644ebc-739f-451b-b8c0-b2fd6852d45a" containerName="maas-api" Sep 10 22:31:17.253040 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.252998 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b644ebc-739f-451b-b8c0-b2fd6852d45a" containerName="maas-api" Sep 10 22:31:17.253040 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.253026 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9a64c50-63e9-4a98-a492-c1e77716838e" containerName="maas-api" Sep 10 22:31:17.253200 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.253034 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9a64c50-63e9-4a98-a492-c1e77716838e" containerName="maas-api" Sep 10 22:31:17.253200 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.253055 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="440f4b25-0d6b-40ad-9f3d-4c3f5ca44037" containerName="maas-api" Sep 10 22:31:17.253200 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.253064 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="440f4b25-0d6b-40ad-9f3d-4c3f5ca44037" containerName="maas-api" Sep 10 22:31:17.253200 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.253077 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fdbdc87-97f2-43ca-b3d6-099653368751" containerName="maas-api" Sep 10 22:31:17.253200 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.253085 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fdbdc87-97f2-43ca-b3d6-099653368751" containerName="maas-api" Sep 10 22:31:17.253200 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.253118 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6e6a121-5946-4069-9b1c-6cfbfd294ee1" containerName="test-curl-7608-adbaf7" Sep 10 22:31:17.253200 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.253126 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6e6a121-5946-4069-9b1c-6cfbfd294ee1" containerName="test-curl-7608-adbaf7" Sep 10 22:31:17.253543 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.253227 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b644ebc-739f-451b-b8c0-b2fd6852d45a" containerName="maas-api" Sep 10 22:31:17.253543 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.253238 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9a64c50-63e9-4a98-a492-c1e77716838e" containerName="maas-api" Sep 10 22:31:17.253543 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.253247 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fdbdc87-97f2-43ca-b3d6-099653368751" containerName="maas-api" Sep 10 22:31:17.253543 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.253255 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="440f4b25-0d6b-40ad-9f3d-4c3f5ca44037" containerName="maas-api" Sep 10 22:31:17.253543 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.253270 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6e6a121-5946-4069-9b1c-6cfbfd294ee1" containerName="test-curl-7608-adbaf7" Sep 10 22:31:17.257110 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.257088 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr" Sep 10 22:31:17.259621 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.259602 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-25241d-serving-cert\"" Sep 10 22:31:17.265926 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.265837 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr"] Sep 10 22:31:17.416223 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.416193 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c87017-df10-484a-83ba-5bfbe124dc22-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr\" (UID: \"80c87017-df10-484a-83ba-5bfbe124dc22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr" Sep 10 22:31:17.416395 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.416252 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz9hv\" (UniqueName: \"kubernetes.io/projected/80c87017-df10-484a-83ba-5bfbe124dc22-kube-api-access-pz9hv\") pod \"maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr\" (UID: \"80c87017-df10-484a-83ba-5bfbe124dc22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr" Sep 10 22:31:17.416395 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.416275 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80c87017-df10-484a-83ba-5bfbe124dc22-maas-api-tls\") pod \"maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr\" (UID: \"80c87017-df10-484a-83ba-5bfbe124dc22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr" Sep 10 22:31:17.416395 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.416384 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80c87017-df10-484a-83ba-5bfbe124dc22-metrics-certs\") pod \"maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr\" (UID: \"80c87017-df10-484a-83ba-5bfbe124dc22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr" Sep 10 22:31:17.517675 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.517589 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80c87017-df10-484a-83ba-5bfbe124dc22-metrics-certs\") pod \"maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr\" (UID: \"80c87017-df10-484a-83ba-5bfbe124dc22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr" Sep 10 22:31:17.517675 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.517663 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c87017-df10-484a-83ba-5bfbe124dc22-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr\" (UID: \"80c87017-df10-484a-83ba-5bfbe124dc22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr" Sep 10 22:31:17.517868 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.517731 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz9hv\" (UniqueName: \"kubernetes.io/projected/80c87017-df10-484a-83ba-5bfbe124dc22-kube-api-access-pz9hv\") pod \"maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr\" (UID: \"80c87017-df10-484a-83ba-5bfbe124dc22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr" Sep 10 22:31:17.517868 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.517765 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80c87017-df10-484a-83ba-5bfbe124dc22-maas-api-tls\") pod \"maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr\" (UID: \"80c87017-df10-484a-83ba-5bfbe124dc22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr" Sep 10 22:31:17.518421 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.518397 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c87017-df10-484a-83ba-5bfbe124dc22-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr\" (UID: \"80c87017-df10-484a-83ba-5bfbe124dc22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr" Sep 10 22:31:17.520059 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.520039 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80c87017-df10-484a-83ba-5bfbe124dc22-metrics-certs\") pod \"maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr\" (UID: \"80c87017-df10-484a-83ba-5bfbe124dc22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr" Sep 10 22:31:17.520187 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.520169 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80c87017-df10-484a-83ba-5bfbe124dc22-maas-api-tls\") pod \"maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr\" (UID: \"80c87017-df10-484a-83ba-5bfbe124dc22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr" Sep 10 22:31:17.525527 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.525507 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz9hv\" (UniqueName: \"kubernetes.io/projected/80c87017-df10-484a-83ba-5bfbe124dc22-kube-api-access-pz9hv\") pod \"maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr\" (UID: \"80c87017-df10-484a-83ba-5bfbe124dc22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr" Sep 10 22:31:17.569149 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.569122 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr" Sep 10 22:31:17.842007 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:17.841974 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr"] Sep 10 22:31:17.843756 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:31:17.843727 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c87017_df10_484a_83ba_5bfbe124dc22.slice/crio-2c2cc7b68a6d394a0a9665c05349919534c3cb46245363b47f9d30a58ded6750 WatchSource:0}: Error finding container 2c2cc7b68a6d394a0a9665c05349919534c3cb46245363b47f9d30a58ded6750: Status 404 returned error can't find the container with id 2c2cc7b68a6d394a0a9665c05349919534c3cb46245363b47f9d30a58ded6750 Sep 10 22:31:18.787603 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:18.787571 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr" event={"ID":"80c87017-df10-484a-83ba-5bfbe124dc22","Type":"ContainerStarted","Data":"a6ef5f316a57862f7846175a1075da0b50c90700742fcfe13b3484232dca6e20"} Sep 10 22:31:18.788013 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:18.787619 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr" event={"ID":"80c87017-df10-484a-83ba-5bfbe124dc22","Type":"ContainerStarted","Data":"2c2cc7b68a6d394a0a9665c05349919534c3cb46245363b47f9d30a58ded6750"} Sep 10 22:31:18.788013 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:18.787654 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr" Sep 10 22:31:18.805739 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:18.805694 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr" podStartSLOduration=1.425797346 podStartE2EDuration="1.805680089s" totalImagesPullingTime="379.882743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:31:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:31:18.805312358 +0000 UTC m=+1580.374802323" watchObservedRunningTime="2026-09-10 22:31:18.805680089 +0000 UTC m=+1580.375170067" Sep 10 22:31:24.798006 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:24.797972 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr" Sep 10 22:31:45.906600 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:45.906563 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78"] Sep 10 22:31:45.910944 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:45.910921 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" Sep 10 22:31:45.913861 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:45.913838 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-9b2092-serving-cert\"" Sep 10 22:31:45.920080 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:45.920049 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78"] Sep 10 22:31:45.987106 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:45.987066 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec953d93-6298-418a-b062-2af2bdb2ec26-metrics-certs\") pod \"maas-api-e2e-ipp-b-9b2092-799b56668d-27p78\" (UID: \"ec953d93-6298-418a-b062-2af2bdb2ec26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" Sep 10 22:31:45.987322 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:45.987150 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec953d93-6298-418a-b062-2af2bdb2ec26-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9b2092-799b56668d-27p78\" (UID: \"ec953d93-6298-418a-b062-2af2bdb2ec26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" Sep 10 22:31:45.987322 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:45.987222 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec953d93-6298-418a-b062-2af2bdb2ec26-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9b2092-799b56668d-27p78\" (UID: \"ec953d93-6298-418a-b062-2af2bdb2ec26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" Sep 10 22:31:45.987322 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:45.987287 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76ptz\" (UniqueName: \"kubernetes.io/projected/ec953d93-6298-418a-b062-2af2bdb2ec26-kube-api-access-76ptz\") pod \"maas-api-e2e-ipp-b-9b2092-799b56668d-27p78\" (UID: \"ec953d93-6298-418a-b062-2af2bdb2ec26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" Sep 10 22:31:46.088842 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:46.088805 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec953d93-6298-418a-b062-2af2bdb2ec26-metrics-certs\") pod \"maas-api-e2e-ipp-b-9b2092-799b56668d-27p78\" (UID: \"ec953d93-6298-418a-b062-2af2bdb2ec26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" Sep 10 22:31:46.089057 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:46.088865 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec953d93-6298-418a-b062-2af2bdb2ec26-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9b2092-799b56668d-27p78\" (UID: \"ec953d93-6298-418a-b062-2af2bdb2ec26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" Sep 10 22:31:46.089057 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:46.088884 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec953d93-6298-418a-b062-2af2bdb2ec26-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9b2092-799b56668d-27p78\" (UID: \"ec953d93-6298-418a-b062-2af2bdb2ec26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" Sep 10 22:31:46.089057 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:46.088947 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76ptz\" (UniqueName: \"kubernetes.io/projected/ec953d93-6298-418a-b062-2af2bdb2ec26-kube-api-access-76ptz\") pod \"maas-api-e2e-ipp-b-9b2092-799b56668d-27p78\" (UID: \"ec953d93-6298-418a-b062-2af2bdb2ec26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" Sep 10 22:31:46.089057 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:31:46.089006 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-serving-cert: secret "maas-api-e2e-ipp-b-9b2092-serving-cert" not found Sep 10 22:31:46.089211 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:31:46.089085 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec953d93-6298-418a-b062-2af2bdb2ec26-maas-api-tls podName:ec953d93-6298-418a-b062-2af2bdb2ec26 nodeName:}" failed. No retries permitted until 2026-09-10 22:31:46.589065264 +0000 UTC m=+1608.158555207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec953d93-6298-418a-b062-2af2bdb2ec26-maas-api-tls") pod "maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" (UID: "ec953d93-6298-418a-b062-2af2bdb2ec26") : secret "maas-api-e2e-ipp-b-9b2092-serving-cert" not found Sep 10 22:31:46.089620 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:46.089596 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec953d93-6298-418a-b062-2af2bdb2ec26-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9b2092-799b56668d-27p78\" (UID: \"ec953d93-6298-418a-b062-2af2bdb2ec26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" Sep 10 22:31:46.091350 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:46.091327 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec953d93-6298-418a-b062-2af2bdb2ec26-metrics-certs\") pod \"maas-api-e2e-ipp-b-9b2092-799b56668d-27p78\" (UID: \"ec953d93-6298-418a-b062-2af2bdb2ec26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" Sep 10 22:31:46.097388 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:46.097368 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76ptz\" (UniqueName: \"kubernetes.io/projected/ec953d93-6298-418a-b062-2af2bdb2ec26-kube-api-access-76ptz\") pod \"maas-api-e2e-ipp-b-9b2092-799b56668d-27p78\" (UID: \"ec953d93-6298-418a-b062-2af2bdb2ec26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" Sep 10 22:31:46.595073 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:46.595036 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec953d93-6298-418a-b062-2af2bdb2ec26-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9b2092-799b56668d-27p78\" (UID: \"ec953d93-6298-418a-b062-2af2bdb2ec26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" Sep 10 22:31:46.597483 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:46.597461 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec953d93-6298-418a-b062-2af2bdb2ec26-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9b2092-799b56668d-27p78\" (UID: \"ec953d93-6298-418a-b062-2af2bdb2ec26\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" Sep 10 22:31:46.825290 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:46.825249 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" Sep 10 22:31:47.104807 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:47.104776 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78"] Sep 10 22:31:47.935871 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:47.935836 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" event={"ID":"ec953d93-6298-418a-b062-2af2bdb2ec26","Type":"ContainerStarted","Data":"21b450edb2bba4ee461da7ce4bd3a827220473664a794b59c0db57dff99f7c81"} Sep 10 22:31:47.935871 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:47.935879 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" event={"ID":"ec953d93-6298-418a-b062-2af2bdb2ec26","Type":"ContainerStarted","Data":"d05a423a96cbad33831234878bcadfefa859c23e91dd1c12adaddf894a744814"} Sep 10 22:31:47.936162 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:47.935954 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" Sep 10 22:31:47.952709 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:47.952638 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" podStartSLOduration=2.5734493819999997 podStartE2EDuration="2.952622609s" totalImagesPullingTime="379.173227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:31:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:31:47.951775455 +0000 UTC m=+1609.521265419" watchObservedRunningTime="2026-09-10 22:31:47.952622609 +0000 UTC m=+1609.522112590" Sep 10 22:31:53.946580 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:31:53.946496 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" Sep 10 22:32:50.007751 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:32:50.007712 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7"] Sep 10 22:32:50.012252 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:32:50.012231 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" Sep 10 22:32:50.015248 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:32:50.015221 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-5567cc-serving-cert\"" Sep 10 22:32:50.023510 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:32:50.023485 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7"] Sep 10 22:32:50.025844 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:32:50.025818 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02b84c99-16d0-44fc-a9e0-5a06a0e28610-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7\" (UID: \"02b84c99-16d0-44fc-a9e0-5a06a0e28610\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" Sep 10 22:32:50.025989 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:32:50.025861 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02b84c99-16d0-44fc-a9e0-5a06a0e28610-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7\" (UID: \"02b84c99-16d0-44fc-a9e0-5a06a0e28610\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" Sep 10 22:32:50.025989 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:32:50.025889 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl76p\" (UniqueName: \"kubernetes.io/projected/02b84c99-16d0-44fc-a9e0-5a06a0e28610-kube-api-access-sl76p\") pod \"maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7\" (UID: \"02b84c99-16d0-44fc-a9e0-5a06a0e28610\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" Sep 10 22:32:50.026106 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:32:50.026054 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b84c99-16d0-44fc-a9e0-5a06a0e28610-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7\" (UID: \"02b84c99-16d0-44fc-a9e0-5a06a0e28610\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" Sep 10 22:32:50.127465 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:32:50.127426 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02b84c99-16d0-44fc-a9e0-5a06a0e28610-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7\" (UID: \"02b84c99-16d0-44fc-a9e0-5a06a0e28610\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" Sep 10 22:32:50.127647 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:32:50.127474 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02b84c99-16d0-44fc-a9e0-5a06a0e28610-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7\" (UID: \"02b84c99-16d0-44fc-a9e0-5a06a0e28610\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" Sep 10 22:32:50.127647 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:32:50.127497 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl76p\" (UniqueName: \"kubernetes.io/projected/02b84c99-16d0-44fc-a9e0-5a06a0e28610-kube-api-access-sl76p\") pod \"maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7\" (UID: \"02b84c99-16d0-44fc-a9e0-5a06a0e28610\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" Sep 10 22:32:50.127647 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:32:50.127528 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b84c99-16d0-44fc-a9e0-5a06a0e28610-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7\" (UID: \"02b84c99-16d0-44fc-a9e0-5a06a0e28610\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" Sep 10 22:32:50.127647 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:32:50.127588 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-serving-cert: secret "maas-api-e2e-ipp-cleanup-5567cc-serving-cert" not found Sep 10 22:32:50.127805 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:32:50.127667 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02b84c99-16d0-44fc-a9e0-5a06a0e28610-maas-api-tls podName:02b84c99-16d0-44fc-a9e0-5a06a0e28610 nodeName:}" failed. No retries permitted until 2026-09-10 22:32:50.627648638 +0000 UTC m=+1672.197138589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/02b84c99-16d0-44fc-a9e0-5a06a0e28610-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" (UID: "02b84c99-16d0-44fc-a9e0-5a06a0e28610") : secret "maas-api-e2e-ipp-cleanup-5567cc-serving-cert" not found Sep 10 22:32:50.128186 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:32:50.128158 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b84c99-16d0-44fc-a9e0-5a06a0e28610-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7\" (UID: \"02b84c99-16d0-44fc-a9e0-5a06a0e28610\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" Sep 10 22:32:50.129860 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:32:50.129834 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02b84c99-16d0-44fc-a9e0-5a06a0e28610-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7\" (UID: \"02b84c99-16d0-44fc-a9e0-5a06a0e28610\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" Sep 10 22:32:50.142761 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:32:50.142743 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl76p\" (UniqueName: \"kubernetes.io/projected/02b84c99-16d0-44fc-a9e0-5a06a0e28610-kube-api-access-sl76p\") pod \"maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7\" (UID: \"02b84c99-16d0-44fc-a9e0-5a06a0e28610\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" Sep 10 22:32:50.632257 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:32:50.632220 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02b84c99-16d0-44fc-a9e0-5a06a0e28610-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7\" (UID: \"02b84c99-16d0-44fc-a9e0-5a06a0e28610\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" Sep 10 22:32:50.634513 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:32:50.634493 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02b84c99-16d0-44fc-a9e0-5a06a0e28610-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7\" (UID: \"02b84c99-16d0-44fc-a9e0-5a06a0e28610\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" Sep 10 22:32:50.924839 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:32:50.924743 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" Sep 10 22:32:51.200367 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:32:51.200340 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7"] Sep 10 22:32:51.202216 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:32:51.202182 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b84c99_16d0_44fc_a9e0_5a06a0e28610.slice/crio-49aa5c9f3c07273167330b5e075b4f2269428a1195aae3805ba8553bb48b115f WatchSource:0}: Error finding container 49aa5c9f3c07273167330b5e075b4f2269428a1195aae3805ba8553bb48b115f: Status 404 returned error can't find the container with id 49aa5c9f3c07273167330b5e075b4f2269428a1195aae3805ba8553bb48b115f Sep 10 22:32:51.255762 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:32:51.255726 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" event={"ID":"02b84c99-16d0-44fc-a9e0-5a06a0e28610","Type":"ContainerStarted","Data":"49aa5c9f3c07273167330b5e075b4f2269428a1195aae3805ba8553bb48b115f"} Sep 10 22:32:52.262603 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:32:52.262565 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" event={"ID":"02b84c99-16d0-44fc-a9e0-5a06a0e28610","Type":"ContainerStarted","Data":"77ff5bf1f13b6fd8b2c6f14ad257008701e94776f3ed9fbc7570021f8e2f4ce7"} Sep 10 22:32:52.263010 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:32:52.262636 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" Sep 10 22:32:52.279012 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:32:52.278967 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" podStartSLOduration=2.927600494 podStartE2EDuration="3.278949652s" totalImagesPullingTime="351.349158ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:32:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:32:52.278286783 +0000 UTC m=+1673.847776750" watchObservedRunningTime="2026-09-10 22:32:52.278949652 +0000 UTC m=+1673.848439608" Sep 10 22:32:58.274919 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:32:58.274876 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" Sep 10 22:33:20.543250 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:20.543203 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7"] Sep 10 22:33:20.543749 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:20.543505 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" podUID="02b84c99-16d0-44fc-a9e0-5a06a0e28610" containerName="maas-api" containerID="cri-o://77ff5bf1f13b6fd8b2c6f14ad257008701e94776f3ed9fbc7570021f8e2f4ce7" gracePeriod=30 Sep 10 22:33:20.795137 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:20.795079 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" Sep 10 22:33:20.829287 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:20.829260 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b84c99-16d0-44fc-a9e0-5a06a0e28610-openshift-service-ca\") pod \"02b84c99-16d0-44fc-a9e0-5a06a0e28610\" (UID: \"02b84c99-16d0-44fc-a9e0-5a06a0e28610\") " Sep 10 22:33:20.829433 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:20.829296 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl76p\" (UniqueName: \"kubernetes.io/projected/02b84c99-16d0-44fc-a9e0-5a06a0e28610-kube-api-access-sl76p\") pod \"02b84c99-16d0-44fc-a9e0-5a06a0e28610\" (UID: \"02b84c99-16d0-44fc-a9e0-5a06a0e28610\") " Sep 10 22:33:20.829433 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:20.829328 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02b84c99-16d0-44fc-a9e0-5a06a0e28610-maas-api-tls\") pod \"02b84c99-16d0-44fc-a9e0-5a06a0e28610\" (UID: \"02b84c99-16d0-44fc-a9e0-5a06a0e28610\") " Sep 10 22:33:20.829632 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:20.829604 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02b84c99-16d0-44fc-a9e0-5a06a0e28610-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "02b84c99-16d0-44fc-a9e0-5a06a0e28610" (UID: "02b84c99-16d0-44fc-a9e0-5a06a0e28610"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:33:20.831361 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:20.831328 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02b84c99-16d0-44fc-a9e0-5a06a0e28610-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "02b84c99-16d0-44fc-a9e0-5a06a0e28610" (UID: "02b84c99-16d0-44fc-a9e0-5a06a0e28610"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:33:20.831460 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:20.831380 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b84c99-16d0-44fc-a9e0-5a06a0e28610-kube-api-access-sl76p" (OuterVolumeSpecName: "kube-api-access-sl76p") pod "02b84c99-16d0-44fc-a9e0-5a06a0e28610" (UID: "02b84c99-16d0-44fc-a9e0-5a06a0e28610"). InnerVolumeSpecName "kube-api-access-sl76p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:33:20.930446 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:20.930419 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02b84c99-16d0-44fc-a9e0-5a06a0e28610-metrics-certs\") pod \"02b84c99-16d0-44fc-a9e0-5a06a0e28610\" (UID: \"02b84c99-16d0-44fc-a9e0-5a06a0e28610\") " Sep 10 22:33:20.930612 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:20.930600 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b84c99-16d0-44fc-a9e0-5a06a0e28610-openshift-service-ca\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:33:20.930656 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:20.930614 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl76p\" (UniqueName: \"kubernetes.io/projected/02b84c99-16d0-44fc-a9e0-5a06a0e28610-kube-api-access-sl76p\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:33:20.930656 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:20.930622 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02b84c99-16d0-44fc-a9e0-5a06a0e28610-maas-api-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:33:20.932350 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:20.932327 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02b84c99-16d0-44fc-a9e0-5a06a0e28610-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "02b84c99-16d0-44fc-a9e0-5a06a0e28610" (UID: "02b84c99-16d0-44fc-a9e0-5a06a0e28610"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:33:21.031632 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:21.031604 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02b84c99-16d0-44fc-a9e0-5a06a0e28610-metrics-certs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:33:21.408203 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:21.408168 2564 generic.go:356] "Generic (PLEG): container finished" podID="02b84c99-16d0-44fc-a9e0-5a06a0e28610" containerID="77ff5bf1f13b6fd8b2c6f14ad257008701e94776f3ed9fbc7570021f8e2f4ce7" exitCode=0 Sep 10 22:33:21.408351 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:21.408209 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" Sep 10 22:33:21.408351 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:21.408252 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" event={"ID":"02b84c99-16d0-44fc-a9e0-5a06a0e28610","Type":"ContainerDied","Data":"77ff5bf1f13b6fd8b2c6f14ad257008701e94776f3ed9fbc7570021f8e2f4ce7"} Sep 10 22:33:21.408351 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:21.408303 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7" event={"ID":"02b84c99-16d0-44fc-a9e0-5a06a0e28610","Type":"ContainerDied","Data":"49aa5c9f3c07273167330b5e075b4f2269428a1195aae3805ba8553bb48b115f"} Sep 10 22:33:21.408351 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:21.408325 2564 scope.go:117] "RemoveContainer" containerID="77ff5bf1f13b6fd8b2c6f14ad257008701e94776f3ed9fbc7570021f8e2f4ce7" Sep 10 22:33:21.417332 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:21.417313 2564 scope.go:117] "RemoveContainer" containerID="77ff5bf1f13b6fd8b2c6f14ad257008701e94776f3ed9fbc7570021f8e2f4ce7" Sep 10 22:33:21.417613 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:33:21.417588 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ff5bf1f13b6fd8b2c6f14ad257008701e94776f3ed9fbc7570021f8e2f4ce7\": container with ID starting with 77ff5bf1f13b6fd8b2c6f14ad257008701e94776f3ed9fbc7570021f8e2f4ce7 not found: ID does not exist" containerID="77ff5bf1f13b6fd8b2c6f14ad257008701e94776f3ed9fbc7570021f8e2f4ce7" Sep 10 22:33:21.417686 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:21.417617 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ff5bf1f13b6fd8b2c6f14ad257008701e94776f3ed9fbc7570021f8e2f4ce7"} err="failed to get container status \"77ff5bf1f13b6fd8b2c6f14ad257008701e94776f3ed9fbc7570021f8e2f4ce7\": rpc error: code = NotFound desc = could not find container \"77ff5bf1f13b6fd8b2c6f14ad257008701e94776f3ed9fbc7570021f8e2f4ce7\": container with ID starting with 77ff5bf1f13b6fd8b2c6f14ad257008701e94776f3ed9fbc7570021f8e2f4ce7 not found: ID does not exist" Sep 10 22:33:21.426883 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:21.426862 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7"] Sep 10 22:33:21.428820 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:21.428801 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5567cc-5644cc7889-nrrj7"] Sep 10 22:33:22.937394 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:22.937361 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02b84c99-16d0-44fc-a9e0-5a06a0e28610" path="/var/lib/kubelet/pods/02b84c99-16d0-44fc-a9e0-5a06a0e28610/volumes" Sep 10 22:33:46.321569 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:46.321536 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr"] Sep 10 22:33:46.322076 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:46.321838 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr" podUID="80c87017-df10-484a-83ba-5bfbe124dc22" containerName="maas-api" containerID="cri-o://a6ef5f316a57862f7846175a1075da0b50c90700742fcfe13b3484232dca6e20" gracePeriod=30 Sep 10 22:33:46.545231 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:46.545199 2564 generic.go:356] "Generic (PLEG): container finished" podID="80c87017-df10-484a-83ba-5bfbe124dc22" containerID="a6ef5f316a57862f7846175a1075da0b50c90700742fcfe13b3484232dca6e20" exitCode=0 Sep 10 22:33:46.545436 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:46.545272 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr" event={"ID":"80c87017-df10-484a-83ba-5bfbe124dc22","Type":"ContainerDied","Data":"a6ef5f316a57862f7846175a1075da0b50c90700742fcfe13b3484232dca6e20"} Sep 10 22:33:46.580244 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:46.580178 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr" Sep 10 22:33:46.657148 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:46.657111 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80c87017-df10-484a-83ba-5bfbe124dc22-maas-api-tls\") pod \"80c87017-df10-484a-83ba-5bfbe124dc22\" (UID: \"80c87017-df10-484a-83ba-5bfbe124dc22\") " Sep 10 22:33:46.657148 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:46.657154 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80c87017-df10-484a-83ba-5bfbe124dc22-metrics-certs\") pod \"80c87017-df10-484a-83ba-5bfbe124dc22\" (UID: \"80c87017-df10-484a-83ba-5bfbe124dc22\") " Sep 10 22:33:46.657403 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:46.657189 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c87017-df10-484a-83ba-5bfbe124dc22-openshift-service-ca\") pod \"80c87017-df10-484a-83ba-5bfbe124dc22\" (UID: \"80c87017-df10-484a-83ba-5bfbe124dc22\") " Sep 10 22:33:46.657403 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:46.657295 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pz9hv\" (UniqueName: \"kubernetes.io/projected/80c87017-df10-484a-83ba-5bfbe124dc22-kube-api-access-pz9hv\") pod \"80c87017-df10-484a-83ba-5bfbe124dc22\" (UID: \"80c87017-df10-484a-83ba-5bfbe124dc22\") " Sep 10 22:33:46.657687 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:46.657655 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c87017-df10-484a-83ba-5bfbe124dc22-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "80c87017-df10-484a-83ba-5bfbe124dc22" (UID: "80c87017-df10-484a-83ba-5bfbe124dc22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:33:46.659365 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:46.659333 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c87017-df10-484a-83ba-5bfbe124dc22-kube-api-access-pz9hv" (OuterVolumeSpecName: "kube-api-access-pz9hv") pod "80c87017-df10-484a-83ba-5bfbe124dc22" (UID: "80c87017-df10-484a-83ba-5bfbe124dc22"). InnerVolumeSpecName "kube-api-access-pz9hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:33:46.659365 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:46.659347 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80c87017-df10-484a-83ba-5bfbe124dc22-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "80c87017-df10-484a-83ba-5bfbe124dc22" (UID: "80c87017-df10-484a-83ba-5bfbe124dc22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:33:46.659543 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:46.659334 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80c87017-df10-484a-83ba-5bfbe124dc22-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "80c87017-df10-484a-83ba-5bfbe124dc22" (UID: "80c87017-df10-484a-83ba-5bfbe124dc22"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:33:46.759094 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:46.759052 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz9hv\" (UniqueName: \"kubernetes.io/projected/80c87017-df10-484a-83ba-5bfbe124dc22-kube-api-access-pz9hv\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:33:46.759094 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:46.759086 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80c87017-df10-484a-83ba-5bfbe124dc22-maas-api-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:33:46.759094 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:46.759095 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80c87017-df10-484a-83ba-5bfbe124dc22-metrics-certs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:33:46.759094 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:46.759103 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c87017-df10-484a-83ba-5bfbe124dc22-openshift-service-ca\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:33:47.552224 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:47.552189 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr" Sep 10 22:33:47.552224 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:47.552200 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr" event={"ID":"80c87017-df10-484a-83ba-5bfbe124dc22","Type":"ContainerDied","Data":"2c2cc7b68a6d394a0a9665c05349919534c3cb46245363b47f9d30a58ded6750"} Sep 10 22:33:47.552740 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:47.552258 2564 scope.go:117] "RemoveContainer" containerID="a6ef5f316a57862f7846175a1075da0b50c90700742fcfe13b3484232dca6e20" Sep 10 22:33:47.570624 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:47.570591 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr"] Sep 10 22:33:47.576545 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:47.576515 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-25241d-7647fd69-b6ssr"] Sep 10 22:33:48.938241 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:33:48.938206 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80c87017-df10-484a-83ba-5bfbe124dc22" path="/var/lib/kubelet/pods/80c87017-df10-484a-83ba-5bfbe124dc22/volumes" Sep 10 22:34:38.804723 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:34:38.804642 2564 generic.go:356] "Generic (PLEG): container finished" podID="6dd29f4d-c5ca-43b3-97d4-130855c684b0" containerID="08feb0675b1cf9d60057841da8d4c421904ff8103b26ed1fe8a111aab819e4f7" exitCode=1 Sep 10 22:34:38.804723 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:34:38.804712 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc" event={"ID":"6dd29f4d-c5ca-43b3-97d4-130855c684b0","Type":"ContainerDied","Data":"08feb0675b1cf9d60057841da8d4c421904ff8103b26ed1fe8a111aab819e4f7"} Sep 10 22:34:38.805244 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:34:38.805027 2564 scope.go:117] "RemoveContainer" containerID="08feb0675b1cf9d60057841da8d4c421904ff8103b26ed1fe8a111aab819e4f7" Sep 10 22:34:38.805326 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:34:38.805312 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 22:34:39.811974 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:34:39.811935 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc" event={"ID":"6dd29f4d-c5ca-43b3-97d4-130855c684b0","Type":"ContainerStarted","Data":"9a700551b5031b32d907fc2cb34fc4178da635806375e1afd01504918288a47e"} Sep 10 22:34:39.812385 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:34:39.812158 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc" Sep 10 22:34:50.867790 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:34:50.867752 2564 generic.go:356] "Generic (PLEG): container finished" podID="6dd29f4d-c5ca-43b3-97d4-130855c684b0" containerID="9a700551b5031b32d907fc2cb34fc4178da635806375e1afd01504918288a47e" exitCode=1 Sep 10 22:34:50.868340 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:34:50.867822 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc" event={"ID":"6dd29f4d-c5ca-43b3-97d4-130855c684b0","Type":"ContainerDied","Data":"9a700551b5031b32d907fc2cb34fc4178da635806375e1afd01504918288a47e"} Sep 10 22:34:50.868340 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:34:50.867929 2564 scope.go:117] "RemoveContainer" containerID="08feb0675b1cf9d60057841da8d4c421904ff8103b26ed1fe8a111aab819e4f7" Sep 10 22:34:50.868340 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:34:50.868274 2564 scope.go:117] "RemoveContainer" containerID="9a700551b5031b32d907fc2cb34fc4178da635806375e1afd01504918288a47e" Sep 10 22:34:50.868536 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:34:50.868513 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-tv2jc_opendatahub(6dd29f4d-c5ca-43b3-97d4-130855c684b0)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc" podUID="6dd29f4d-c5ca-43b3-97d4-130855c684b0" Sep 10 22:34:56.964493 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:34:56.964449 2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc" Sep 10 22:34:56.964888 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:34:56.964846 2564 scope.go:117] "RemoveContainer" containerID="9a700551b5031b32d907fc2cb34fc4178da635806375e1afd01504918288a47e" Sep 10 22:34:56.965083 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:34:56.965064 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-tv2jc_opendatahub(6dd29f4d-c5ca-43b3-97d4-130855c684b0)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc" podUID="6dd29f4d-c5ca-43b3-97d4-130855c684b0" Sep 10 22:35:10.936819 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:10.936793 2564 scope.go:117] "RemoveContainer" containerID="9a700551b5031b32d907fc2cb34fc4178da635806375e1afd01504918288a47e" Sep 10 22:35:11.977071 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:11.977040 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc" event={"ID":"6dd29f4d-c5ca-43b3-97d4-130855c684b0","Type":"ContainerStarted","Data":"43b2dd99da6fd1561a44659e0b9d7716d1865d54d89015091aa6e753495a44b8"} Sep 10 22:35:11.977529 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:11.977238 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc" Sep 10 22:35:28.119838 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:28.119801 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78"] Sep 10 22:35:28.120434 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:28.120139 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" podUID="ec953d93-6298-418a-b062-2af2bdb2ec26" containerName="maas-api" containerID="cri-o://21b450edb2bba4ee461da7ce4bd3a827220473664a794b59c0db57dff99f7c81" gracePeriod=30 Sep 10 22:35:28.369430 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:28.369406 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" Sep 10 22:35:28.490953 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:28.490850 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec953d93-6298-418a-b062-2af2bdb2ec26-openshift-service-ca\") pod \"ec953d93-6298-418a-b062-2af2bdb2ec26\" (UID: \"ec953d93-6298-418a-b062-2af2bdb2ec26\") " Sep 10 22:35:28.490953 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:28.490891 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec953d93-6298-418a-b062-2af2bdb2ec26-maas-api-tls\") pod \"ec953d93-6298-418a-b062-2af2bdb2ec26\" (UID: \"ec953d93-6298-418a-b062-2af2bdb2ec26\") " Sep 10 22:35:28.491196 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:28.491032 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec953d93-6298-418a-b062-2af2bdb2ec26-metrics-certs\") pod \"ec953d93-6298-418a-b062-2af2bdb2ec26\" (UID: \"ec953d93-6298-418a-b062-2af2bdb2ec26\") " Sep 10 22:35:28.491196 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:28.491101 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76ptz\" (UniqueName: \"kubernetes.io/projected/ec953d93-6298-418a-b062-2af2bdb2ec26-kube-api-access-76ptz\") pod \"ec953d93-6298-418a-b062-2af2bdb2ec26\" (UID: \"ec953d93-6298-418a-b062-2af2bdb2ec26\") " Sep 10 22:35:28.491303 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:28.491283 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec953d93-6298-418a-b062-2af2bdb2ec26-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ec953d93-6298-418a-b062-2af2bdb2ec26" (UID: "ec953d93-6298-418a-b062-2af2bdb2ec26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:35:28.491482 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:28.491462 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec953d93-6298-418a-b062-2af2bdb2ec26-openshift-service-ca\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:35:28.493202 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:28.493172 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec953d93-6298-418a-b062-2af2bdb2ec26-kube-api-access-76ptz" (OuterVolumeSpecName: "kube-api-access-76ptz") pod "ec953d93-6298-418a-b062-2af2bdb2ec26" (UID: "ec953d93-6298-418a-b062-2af2bdb2ec26"). InnerVolumeSpecName "kube-api-access-76ptz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:35:28.493296 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:28.493217 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec953d93-6298-418a-b062-2af2bdb2ec26-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ec953d93-6298-418a-b062-2af2bdb2ec26" (UID: "ec953d93-6298-418a-b062-2af2bdb2ec26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:35:28.493296 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:28.493227 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec953d93-6298-418a-b062-2af2bdb2ec26-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ec953d93-6298-418a-b062-2af2bdb2ec26" (UID: "ec953d93-6298-418a-b062-2af2bdb2ec26"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:35:28.592340 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:28.592305 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76ptz\" (UniqueName: \"kubernetes.io/projected/ec953d93-6298-418a-b062-2af2bdb2ec26-kube-api-access-76ptz\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:35:28.592340 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:28.592333 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec953d93-6298-418a-b062-2af2bdb2ec26-maas-api-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:35:28.592340 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:28.592342 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec953d93-6298-418a-b062-2af2bdb2ec26-metrics-certs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:35:29.062013 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:29.061975 2564 generic.go:356] "Generic (PLEG): container finished" podID="ec953d93-6298-418a-b062-2af2bdb2ec26" containerID="21b450edb2bba4ee461da7ce4bd3a827220473664a794b59c0db57dff99f7c81" exitCode=0 Sep 10 22:35:29.062183 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:29.062026 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" Sep 10 22:35:29.062183 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:29.062069 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" event={"ID":"ec953d93-6298-418a-b062-2af2bdb2ec26","Type":"ContainerDied","Data":"21b450edb2bba4ee461da7ce4bd3a827220473664a794b59c0db57dff99f7c81"} Sep 10 22:35:29.062183 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:29.062134 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78" event={"ID":"ec953d93-6298-418a-b062-2af2bdb2ec26","Type":"ContainerDied","Data":"d05a423a96cbad33831234878bcadfefa859c23e91dd1c12adaddf894a744814"} Sep 10 22:35:29.062183 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:29.062157 2564 scope.go:117] "RemoveContainer" containerID="21b450edb2bba4ee461da7ce4bd3a827220473664a794b59c0db57dff99f7c81" Sep 10 22:35:29.076266 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:29.076237 2564 scope.go:117] "RemoveContainer" containerID="21b450edb2bba4ee461da7ce4bd3a827220473664a794b59c0db57dff99f7c81" Sep 10 22:35:29.076584 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:35:29.076558 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21b450edb2bba4ee461da7ce4bd3a827220473664a794b59c0db57dff99f7c81\": container with ID starting with 21b450edb2bba4ee461da7ce4bd3a827220473664a794b59c0db57dff99f7c81 not found: ID does not exist" containerID="21b450edb2bba4ee461da7ce4bd3a827220473664a794b59c0db57dff99f7c81" Sep 10 22:35:29.076690 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:29.076595 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b450edb2bba4ee461da7ce4bd3a827220473664a794b59c0db57dff99f7c81"} err="failed to get container status \"21b450edb2bba4ee461da7ce4bd3a827220473664a794b59c0db57dff99f7c81\": rpc error: code = NotFound desc = could not find container \"21b450edb2bba4ee461da7ce4bd3a827220473664a794b59c0db57dff99f7c81\": container with ID starting with 21b450edb2bba4ee461da7ce4bd3a827220473664a794b59c0db57dff99f7c81 not found: ID does not exist" Sep 10 22:35:29.081989 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:29.081965 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78"] Sep 10 22:35:29.083842 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:29.083821 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b2092-799b56668d-27p78"] Sep 10 22:35:30.937224 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:30.937188 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec953d93-6298-418a-b062-2af2bdb2ec26" path="/var/lib/kubelet/pods/ec953d93-6298-418a-b062-2af2bdb2ec26/volumes" Sep 10 22:35:42.983553 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:42.983522 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tv2jc" Sep 10 22:35:45.424266 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:45.424232 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw"] Sep 10 22:35:45.424750 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:45.424449 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" podUID="8de8a0cd-6f57-4778-b376-b64ca7e91d0c" containerName="maas-api" containerID="cri-o://766c3ea70814915d15ca0ad545e1e2034ffc542b7a16b2ddc28427f97af967f6" gracePeriod=30 Sep 10 22:35:45.677316 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:45.677243 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" Sep 10 22:35:45.759292 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:45.759261 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-maas-api-tls\") pod \"8de8a0cd-6f57-4778-b376-b64ca7e91d0c\" (UID: \"8de8a0cd-6f57-4778-b376-b64ca7e91d0c\") " Sep 10 22:35:45.759485 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:45.759306 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-openshift-service-ca\") pod \"8de8a0cd-6f57-4778-b376-b64ca7e91d0c\" (UID: \"8de8a0cd-6f57-4778-b376-b64ca7e91d0c\") " Sep 10 22:35:45.759485 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:45.759337 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-metrics-certs\") pod \"8de8a0cd-6f57-4778-b376-b64ca7e91d0c\" (UID: \"8de8a0cd-6f57-4778-b376-b64ca7e91d0c\") " Sep 10 22:35:45.759485 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:45.759409 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2flc7\" (UniqueName: \"kubernetes.io/projected/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-kube-api-access-2flc7\") pod \"8de8a0cd-6f57-4778-b376-b64ca7e91d0c\" (UID: \"8de8a0cd-6f57-4778-b376-b64ca7e91d0c\") " Sep 10 22:35:45.760148 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:45.760110 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8de8a0cd-6f57-4778-b376-b64ca7e91d0c" (UID: "8de8a0cd-6f57-4778-b376-b64ca7e91d0c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:35:45.761664 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:45.761642 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8de8a0cd-6f57-4778-b376-b64ca7e91d0c" (UID: "8de8a0cd-6f57-4778-b376-b64ca7e91d0c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:35:45.762290 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:45.762258 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8de8a0cd-6f57-4778-b376-b64ca7e91d0c" (UID: "8de8a0cd-6f57-4778-b376-b64ca7e91d0c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:35:45.762387 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:45.762281 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-kube-api-access-2flc7" (OuterVolumeSpecName: "kube-api-access-2flc7") pod "8de8a0cd-6f57-4778-b376-b64ca7e91d0c" (UID: "8de8a0cd-6f57-4778-b376-b64ca7e91d0c"). InnerVolumeSpecName "kube-api-access-2flc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:35:45.861216 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:45.861173 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-maas-api-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:35:45.861216 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:45.861209 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-openshift-service-ca\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:35:45.861216 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:45.861221 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-metrics-certs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:35:45.861471 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:45.861232 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2flc7\" (UniqueName: \"kubernetes.io/projected/8de8a0cd-6f57-4778-b376-b64ca7e91d0c-kube-api-access-2flc7\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:35:46.146896 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:46.146863 2564 generic.go:356] "Generic (PLEG): container finished" podID="8de8a0cd-6f57-4778-b376-b64ca7e91d0c" containerID="766c3ea70814915d15ca0ad545e1e2034ffc542b7a16b2ddc28427f97af967f6" exitCode=0 Sep 10 22:35:46.147094 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:46.146936 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" Sep 10 22:35:46.147094 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:46.146950 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" event={"ID":"8de8a0cd-6f57-4778-b376-b64ca7e91d0c","Type":"ContainerDied","Data":"766c3ea70814915d15ca0ad545e1e2034ffc542b7a16b2ddc28427f97af967f6"} Sep 10 22:35:46.147094 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:46.147031 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw" event={"ID":"8de8a0cd-6f57-4778-b376-b64ca7e91d0c","Type":"ContainerDied","Data":"44741d41ce7488c25c7d8beaab228f9d4bfcc7b749de39bd02151e6b62099a8f"} Sep 10 22:35:46.147094 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:46.147063 2564 scope.go:117] "RemoveContainer" containerID="766c3ea70814915d15ca0ad545e1e2034ffc542b7a16b2ddc28427f97af967f6" Sep 10 22:35:46.162347 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:46.161984 2564 scope.go:117] "RemoveContainer" containerID="766c3ea70814915d15ca0ad545e1e2034ffc542b7a16b2ddc28427f97af967f6" Sep 10 22:35:46.162546 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:35:46.162515 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"766c3ea70814915d15ca0ad545e1e2034ffc542b7a16b2ddc28427f97af967f6\": container with ID starting with 766c3ea70814915d15ca0ad545e1e2034ffc542b7a16b2ddc28427f97af967f6 not found: ID does not exist" containerID="766c3ea70814915d15ca0ad545e1e2034ffc542b7a16b2ddc28427f97af967f6" Sep 10 22:35:46.162716 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:46.162689 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"766c3ea70814915d15ca0ad545e1e2034ffc542b7a16b2ddc28427f97af967f6"} err="failed to get container status \"766c3ea70814915d15ca0ad545e1e2034ffc542b7a16b2ddc28427f97af967f6\": rpc error: code = NotFound desc = could not find container \"766c3ea70814915d15ca0ad545e1e2034ffc542b7a16b2ddc28427f97af967f6\": container with ID starting with 766c3ea70814915d15ca0ad545e1e2034ffc542b7a16b2ddc28427f97af967f6 not found: ID does not exist" Sep 10 22:35:46.185785 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:46.185759 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw"] Sep 10 22:35:46.189971 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:46.189948 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e367c1-ff859f64c-v7wbw"] Sep 10 22:35:46.937038 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:35:46.936996 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8de8a0cd-6f57-4778-b376-b64ca7e91d0c" path="/var/lib/kubelet/pods/8de8a0cd-6f57-4778-b376-b64ca7e91d0c/volumes" Sep 10 22:36:03.327299 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:03.327222 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8"] Sep 10 22:36:03.327877 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:03.327599 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8" podUID="ab5c1a77-830d-463c-a6c2-a1965ee9ffad" containerName="maas-api" containerID="cri-o://74064813a2bb82a1a01bb77cab76f999dbfade7dfdf05e92d5d8c2da448d3907" gracePeriod=30 Sep 10 22:36:03.573246 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:03.573220 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8" Sep 10 22:36:03.742802 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:03.742774 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5c1a77-830d-463c-a6c2-a1965ee9ffad-maas-api-tls\") pod \"ab5c1a77-830d-463c-a6c2-a1965ee9ffad\" (UID: \"ab5c1a77-830d-463c-a6c2-a1965ee9ffad\") " Sep 10 22:36:03.743012 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:03.742841 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5c1a77-830d-463c-a6c2-a1965ee9ffad-openshift-service-ca\") pod \"ab5c1a77-830d-463c-a6c2-a1965ee9ffad\" (UID: \"ab5c1a77-830d-463c-a6c2-a1965ee9ffad\") " Sep 10 22:36:03.743012 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:03.742871 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcgkv\" (UniqueName: \"kubernetes.io/projected/ab5c1a77-830d-463c-a6c2-a1965ee9ffad-kube-api-access-rcgkv\") pod \"ab5c1a77-830d-463c-a6c2-a1965ee9ffad\" (UID: \"ab5c1a77-830d-463c-a6c2-a1965ee9ffad\") " Sep 10 22:36:03.743012 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:03.742923 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab5c1a77-830d-463c-a6c2-a1965ee9ffad-metrics-certs\") pod \"ab5c1a77-830d-463c-a6c2-a1965ee9ffad\" (UID: \"ab5c1a77-830d-463c-a6c2-a1965ee9ffad\") " Sep 10 22:36:03.743272 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:03.743247 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5c1a77-830d-463c-a6c2-a1965ee9ffad-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ab5c1a77-830d-463c-a6c2-a1965ee9ffad" (UID: "ab5c1a77-830d-463c-a6c2-a1965ee9ffad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:36:03.744884 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:03.744859 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab5c1a77-830d-463c-a6c2-a1965ee9ffad-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ab5c1a77-830d-463c-a6c2-a1965ee9ffad" (UID: "ab5c1a77-830d-463c-a6c2-a1965ee9ffad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:36:03.745013 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:03.744880 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab5c1a77-830d-463c-a6c2-a1965ee9ffad-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ab5c1a77-830d-463c-a6c2-a1965ee9ffad" (UID: "ab5c1a77-830d-463c-a6c2-a1965ee9ffad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:36:03.745013 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:03.744931 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5c1a77-830d-463c-a6c2-a1965ee9ffad-kube-api-access-rcgkv" (OuterVolumeSpecName: "kube-api-access-rcgkv") pod "ab5c1a77-830d-463c-a6c2-a1965ee9ffad" (UID: "ab5c1a77-830d-463c-a6c2-a1965ee9ffad"). InnerVolumeSpecName "kube-api-access-rcgkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:36:03.844590 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:03.844561 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5c1a77-830d-463c-a6c2-a1965ee9ffad-maas-api-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:36:03.844590 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:03.844587 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5c1a77-830d-463c-a6c2-a1965ee9ffad-openshift-service-ca\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:36:03.844590 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:03.844598 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcgkv\" (UniqueName: \"kubernetes.io/projected/ab5c1a77-830d-463c-a6c2-a1965ee9ffad-kube-api-access-rcgkv\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:36:03.844796 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:03.844606 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab5c1a77-830d-463c-a6c2-a1965ee9ffad-metrics-certs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:36:04.241735 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:04.241702 2564 generic.go:356] "Generic (PLEG): container finished" podID="ab5c1a77-830d-463c-a6c2-a1965ee9ffad" containerID="74064813a2bb82a1a01bb77cab76f999dbfade7dfdf05e92d5d8c2da448d3907" exitCode=0 Sep 10 22:36:04.241920 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:04.241748 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8" Sep 10 22:36:04.241920 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:04.241790 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8" event={"ID":"ab5c1a77-830d-463c-a6c2-a1965ee9ffad","Type":"ContainerDied","Data":"74064813a2bb82a1a01bb77cab76f999dbfade7dfdf05e92d5d8c2da448d3907"} Sep 10 22:36:04.241920 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:04.241841 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8" event={"ID":"ab5c1a77-830d-463c-a6c2-a1965ee9ffad","Type":"ContainerDied","Data":"608e41df7ac3130054e2a33e72ce227c20e245c6aef8ababa7146efd7a9b25e9"} Sep 10 22:36:04.241920 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:04.241862 2564 scope.go:117] "RemoveContainer" containerID="74064813a2bb82a1a01bb77cab76f999dbfade7dfdf05e92d5d8c2da448d3907" Sep 10 22:36:04.250678 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:04.250658 2564 scope.go:117] "RemoveContainer" containerID="74064813a2bb82a1a01bb77cab76f999dbfade7dfdf05e92d5d8c2da448d3907" Sep 10 22:36:04.250994 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:36:04.250974 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74064813a2bb82a1a01bb77cab76f999dbfade7dfdf05e92d5d8c2da448d3907\": container with ID starting with 74064813a2bb82a1a01bb77cab76f999dbfade7dfdf05e92d5d8c2da448d3907 not found: ID does not exist" containerID="74064813a2bb82a1a01bb77cab76f999dbfade7dfdf05e92d5d8c2da448d3907" Sep 10 22:36:04.251053 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:04.251003 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74064813a2bb82a1a01bb77cab76f999dbfade7dfdf05e92d5d8c2da448d3907"} err="failed to get container status \"74064813a2bb82a1a01bb77cab76f999dbfade7dfdf05e92d5d8c2da448d3907\": rpc error: code = NotFound desc = could not find container \"74064813a2bb82a1a01bb77cab76f999dbfade7dfdf05e92d5d8c2da448d3907\": container with ID starting with 74064813a2bb82a1a01bb77cab76f999dbfade7dfdf05e92d5d8c2da448d3907 not found: ID does not exist" Sep 10 22:36:04.263536 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:04.263514 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8"] Sep 10 22:36:04.272676 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:04.272656 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0ada08-5bc77d64d9-j6bg8"] Sep 10 22:36:04.936522 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:04.936491 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab5c1a77-830d-463c-a6c2-a1965ee9ffad" path="/var/lib/kubelet/pods/ab5c1a77-830d-463c-a6c2-a1965ee9ffad/volumes" Sep 10 22:36:13.080468 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:13.080436 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b6f6f558b-jc7ct"] Sep 10 22:36:13.080886 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:13.080627 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5b6f6f558b-jc7ct" podUID="27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0" containerName="manager" containerID="cri-o://38fe75a95ad594703011c4a447dfd36aad0221266668d0738eac2974e88f1392" gracePeriod=30 Sep 10 22:36:13.295819 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:13.295783 2564 generic.go:356] "Generic (PLEG): container finished" podID="27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0" containerID="38fe75a95ad594703011c4a447dfd36aad0221266668d0738eac2974e88f1392" exitCode=0 Sep 10 22:36:13.295989 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:13.295848 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b6f6f558b-jc7ct" event={"ID":"27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0","Type":"ContainerDied","Data":"38fe75a95ad594703011c4a447dfd36aad0221266668d0738eac2974e88f1392"} Sep 10 22:36:13.328055 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:13.328030 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b6f6f558b-jc7ct" Sep 10 22:36:13.426596 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:13.426566 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0-metrics-certs\") pod \"27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0\" (UID: \"27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0\") " Sep 10 22:36:13.426776 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:13.426644 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99c7w\" (UniqueName: \"kubernetes.io/projected/27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0-kube-api-access-99c7w\") pod \"27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0\" (UID: \"27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0\") " Sep 10 22:36:13.426776 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:13.426739 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0-webhook-cert\") pod \"27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0\" (UID: \"27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0\") " Sep 10 22:36:13.428779 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:13.428748 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0" (UID: "27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:36:13.428779 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:13.428771 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0" (UID: "27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:36:13.428961 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:13.428807 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0-kube-api-access-99c7w" (OuterVolumeSpecName: "kube-api-access-99c7w") pod "27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0" (UID: "27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0"). InnerVolumeSpecName "kube-api-access-99c7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:36:13.528370 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:13.528333 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99c7w\" (UniqueName: \"kubernetes.io/projected/27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0-kube-api-access-99c7w\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:36:13.528370 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:13.528366 2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0-webhook-cert\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:36:13.528370 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:13.528377 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0-metrics-certs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:36:14.302484 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:14.302448 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b6f6f558b-jc7ct" event={"ID":"27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0","Type":"ContainerDied","Data":"fe18a37fe06c4a5169a892eb66c0a1df9097d8ead98a2620e23656396e3b4b37"} Sep 10 22:36:14.302484 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:14.302468 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b6f6f558b-jc7ct" Sep 10 22:36:14.303082 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:14.302519 2564 scope.go:117] "RemoveContainer" containerID="38fe75a95ad594703011c4a447dfd36aad0221266668d0738eac2974e88f1392" Sep 10 22:36:14.324496 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:14.324473 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b6f6f558b-jc7ct"] Sep 10 22:36:14.328457 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:14.328436 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5b6f6f558b-jc7ct"] Sep 10 22:36:14.936892 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:36:14.936861 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0" path="/var/lib/kubelet/pods/27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0/volumes" Sep 10 22:37:15.138020 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.135678 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b6f6f558b-jrdwd"] Sep 10 22:37:15.138020 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.136524 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02b84c99-16d0-44fc-a9e0-5a06a0e28610" containerName="maas-api" Sep 10 22:37:15.138020 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.136547 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="02b84c99-16d0-44fc-a9e0-5a06a0e28610" containerName="maas-api" Sep 10 22:37:15.138020 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.136561 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab5c1a77-830d-463c-a6c2-a1965ee9ffad" containerName="maas-api" Sep 10 22:37:15.138020 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.136570 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab5c1a77-830d-463c-a6c2-a1965ee9ffad" containerName="maas-api" Sep 10 22:37:15.138020 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.136593 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80c87017-df10-484a-83ba-5bfbe124dc22" containerName="maas-api" Sep 10 22:37:15.138020 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.136602 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="80c87017-df10-484a-83ba-5bfbe124dc22" containerName="maas-api" Sep 10 22:37:15.138020 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.136627 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0" containerName="manager" Sep 10 22:37:15.138020 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.136634 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0" containerName="manager" Sep 10 22:37:15.138020 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.136657 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec953d93-6298-418a-b062-2af2bdb2ec26" containerName="maas-api" Sep 10 22:37:15.138020 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.136665 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec953d93-6298-418a-b062-2af2bdb2ec26" containerName="maas-api" Sep 10 22:37:15.138020 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.136684 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de8a0cd-6f57-4778-b376-b64ca7e91d0c" containerName="maas-api" Sep 10 22:37:15.138020 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.136693 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de8a0cd-6f57-4778-b376-b64ca7e91d0c" containerName="maas-api" Sep 10 22:37:15.138020 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.136841 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="27f3569c-9ada-48fd-9e3d-bc5c7bbcabc0" containerName="manager" Sep 10 22:37:15.138020 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.136853 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="02b84c99-16d0-44fc-a9e0-5a06a0e28610" containerName="maas-api" Sep 10 22:37:15.138020 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.136867 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="80c87017-df10-484a-83ba-5bfbe124dc22" containerName="maas-api" Sep 10 22:37:15.138020 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.136875 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec953d93-6298-418a-b062-2af2bdb2ec26" containerName="maas-api" Sep 10 22:37:15.138020 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.136885 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab5c1a77-830d-463c-a6c2-a1965ee9ffad" containerName="maas-api" Sep 10 22:37:15.138020 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.136920 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8de8a0cd-6f57-4778-b376-b64ca7e91d0c" containerName="maas-api" Sep 10 22:37:15.142252 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.142223 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b6f6f558b-jrdwd" Sep 10 22:37:15.145297 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.145273 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zr6dm\"" Sep 10 22:37:15.146799 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.146726 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 22:37:15.146799 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.146741 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 22:37:15.146799 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.146757 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 22:37:15.148949 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.148929 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b6f6f558b-jrdwd"] Sep 10 22:37:15.202326 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.202298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f756t\" (UniqueName: \"kubernetes.io/projected/6c9d6a1d-d6fa-4ccd-958b-01997968122b-kube-api-access-f756t\") pod \"maas-controller-5b6f6f558b-jrdwd\" (UID: \"6c9d6a1d-d6fa-4ccd-958b-01997968122b\") " pod="opendatahub/maas-controller-5b6f6f558b-jrdwd" Sep 10 22:37:15.202455 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.202356 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9d6a1d-d6fa-4ccd-958b-01997968122b-metrics-certs\") pod \"maas-controller-5b6f6f558b-jrdwd\" (UID: \"6c9d6a1d-d6fa-4ccd-958b-01997968122b\") " pod="opendatahub/maas-controller-5b6f6f558b-jrdwd" Sep 10 22:37:15.202455 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.202398 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6c9d6a1d-d6fa-4ccd-958b-01997968122b-webhook-cert\") pod \"maas-controller-5b6f6f558b-jrdwd\" (UID: \"6c9d6a1d-d6fa-4ccd-958b-01997968122b\") " pod="opendatahub/maas-controller-5b6f6f558b-jrdwd" Sep 10 22:37:15.302909 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.302861 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9d6a1d-d6fa-4ccd-958b-01997968122b-metrics-certs\") pod \"maas-controller-5b6f6f558b-jrdwd\" (UID: \"6c9d6a1d-d6fa-4ccd-958b-01997968122b\") " pod="opendatahub/maas-controller-5b6f6f558b-jrdwd" Sep 10 22:37:15.303072 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.302970 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6c9d6a1d-d6fa-4ccd-958b-01997968122b-webhook-cert\") pod \"maas-controller-5b6f6f558b-jrdwd\" (UID: \"6c9d6a1d-d6fa-4ccd-958b-01997968122b\") " pod="opendatahub/maas-controller-5b6f6f558b-jrdwd" Sep 10 22:37:15.303072 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.303044 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f756t\" (UniqueName: \"kubernetes.io/projected/6c9d6a1d-d6fa-4ccd-958b-01997968122b-kube-api-access-f756t\") pod \"maas-controller-5b6f6f558b-jrdwd\" (UID: \"6c9d6a1d-d6fa-4ccd-958b-01997968122b\") " pod="opendatahub/maas-controller-5b6f6f558b-jrdwd" Sep 10 22:37:15.305461 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.305436 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9d6a1d-d6fa-4ccd-958b-01997968122b-metrics-certs\") pod \"maas-controller-5b6f6f558b-jrdwd\" (UID: \"6c9d6a1d-d6fa-4ccd-958b-01997968122b\") " pod="opendatahub/maas-controller-5b6f6f558b-jrdwd" Sep 10 22:37:15.305575 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.305466 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6c9d6a1d-d6fa-4ccd-958b-01997968122b-webhook-cert\") pod \"maas-controller-5b6f6f558b-jrdwd\" (UID: \"6c9d6a1d-d6fa-4ccd-958b-01997968122b\") " pod="opendatahub/maas-controller-5b6f6f558b-jrdwd" Sep 10 22:37:15.311654 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.311629 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f756t\" (UniqueName: \"kubernetes.io/projected/6c9d6a1d-d6fa-4ccd-958b-01997968122b-kube-api-access-f756t\") pod \"maas-controller-5b6f6f558b-jrdwd\" (UID: \"6c9d6a1d-d6fa-4ccd-958b-01997968122b\") " pod="opendatahub/maas-controller-5b6f6f558b-jrdwd" Sep 10 22:37:15.454759 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.454679 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b6f6f558b-jrdwd" Sep 10 22:37:15.728335 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:15.728311 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b6f6f558b-jrdwd"] Sep 10 22:37:15.730702 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:37:15.730673 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c9d6a1d_d6fa_4ccd_958b_01997968122b.slice/crio-f38d6737279b4295724df3bd599e2296d4261692c17793fd159fce2904fbbb6f WatchSource:0}: Error finding container f38d6737279b4295724df3bd599e2296d4261692c17793fd159fce2904fbbb6f: Status 404 returned error can't find the container with id f38d6737279b4295724df3bd599e2296d4261692c17793fd159fce2904fbbb6f Sep 10 22:37:16.610547 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:16.610511 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b6f6f558b-jrdwd" event={"ID":"6c9d6a1d-d6fa-4ccd-958b-01997968122b","Type":"ContainerStarted","Data":"509295bdfb24826b7c69827477a305a454cc8149627cb1e54fdf193d965280e6"} Sep 10 22:37:16.610949 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:16.610570 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b6f6f558b-jrdwd" event={"ID":"6c9d6a1d-d6fa-4ccd-958b-01997968122b","Type":"ContainerStarted","Data":"f38d6737279b4295724df3bd599e2296d4261692c17793fd159fce2904fbbb6f"} Sep 10 22:37:16.610949 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:16.610603 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b6f6f558b-jrdwd" Sep 10 22:37:16.631494 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:16.631452 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b6f6f558b-jrdwd" podStartSLOduration=1.064823509 podStartE2EDuration="1.631437331s" totalImagesPullingTime="566.613822ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:37:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:37:16.629532704 +0000 UTC m=+1938.199022662" watchObservedRunningTime="2026-09-10 22:37:16.631437331 +0000 UTC m=+1938.200927340" Sep 10 22:37:27.617798 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:27.617769 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b6f6f558b-jrdwd" Sep 10 22:37:37.081716 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:37.081682 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b6f6f558b-jrdwd"] Sep 10 22:37:37.082159 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:37.081863 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5b6f6f558b-jrdwd" podUID="6c9d6a1d-d6fa-4ccd-958b-01997968122b" containerName="manager" containerID="cri-o://509295bdfb24826b7c69827477a305a454cc8149627cb1e54fdf193d965280e6" gracePeriod=30 Sep 10 22:37:37.328206 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:37.328179 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b6f6f558b-jrdwd" Sep 10 22:37:37.489318 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:37.489286 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f756t\" (UniqueName: \"kubernetes.io/projected/6c9d6a1d-d6fa-4ccd-958b-01997968122b-kube-api-access-f756t\") pod \"6c9d6a1d-d6fa-4ccd-958b-01997968122b\" (UID: \"6c9d6a1d-d6fa-4ccd-958b-01997968122b\") " Sep 10 22:37:37.489530 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:37.489355 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9d6a1d-d6fa-4ccd-958b-01997968122b-metrics-certs\") pod \"6c9d6a1d-d6fa-4ccd-958b-01997968122b\" (UID: \"6c9d6a1d-d6fa-4ccd-958b-01997968122b\") " Sep 10 22:37:37.489530 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:37.489386 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6c9d6a1d-d6fa-4ccd-958b-01997968122b-webhook-cert\") pod \"6c9d6a1d-d6fa-4ccd-958b-01997968122b\" (UID: \"6c9d6a1d-d6fa-4ccd-958b-01997968122b\") " Sep 10 22:37:37.491454 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:37.491421 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c9d6a1d-d6fa-4ccd-958b-01997968122b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6c9d6a1d-d6fa-4ccd-958b-01997968122b" (UID: "6c9d6a1d-d6fa-4ccd-958b-01997968122b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:37:37.491454 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:37.491438 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c9d6a1d-d6fa-4ccd-958b-01997968122b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6c9d6a1d-d6fa-4ccd-958b-01997968122b" (UID: "6c9d6a1d-d6fa-4ccd-958b-01997968122b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:37:37.491596 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:37.491522 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9d6a1d-d6fa-4ccd-958b-01997968122b-kube-api-access-f756t" (OuterVolumeSpecName: "kube-api-access-f756t") pod "6c9d6a1d-d6fa-4ccd-958b-01997968122b" (UID: "6c9d6a1d-d6fa-4ccd-958b-01997968122b"). InnerVolumeSpecName "kube-api-access-f756t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:37:37.590367 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:37.590325 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f756t\" (UniqueName: \"kubernetes.io/projected/6c9d6a1d-d6fa-4ccd-958b-01997968122b-kube-api-access-f756t\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:37:37.590367 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:37.590363 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9d6a1d-d6fa-4ccd-958b-01997968122b-metrics-certs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:37:37.590615 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:37.590379 2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6c9d6a1d-d6fa-4ccd-958b-01997968122b-webhook-cert\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:37:37.714484 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:37.714446 2564 generic.go:356] "Generic (PLEG): container finished" podID="6c9d6a1d-d6fa-4ccd-958b-01997968122b" containerID="509295bdfb24826b7c69827477a305a454cc8149627cb1e54fdf193d965280e6" exitCode=0 Sep 10 22:37:37.714681 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:37.714495 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b6f6f558b-jrdwd" Sep 10 22:37:37.714681 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:37.714531 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b6f6f558b-jrdwd" event={"ID":"6c9d6a1d-d6fa-4ccd-958b-01997968122b","Type":"ContainerDied","Data":"509295bdfb24826b7c69827477a305a454cc8149627cb1e54fdf193d965280e6"} Sep 10 22:37:37.714681 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:37.714584 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b6f6f558b-jrdwd" event={"ID":"6c9d6a1d-d6fa-4ccd-958b-01997968122b","Type":"ContainerDied","Data":"f38d6737279b4295724df3bd599e2296d4261692c17793fd159fce2904fbbb6f"} Sep 10 22:37:37.714681 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:37.714610 2564 scope.go:117] "RemoveContainer" containerID="509295bdfb24826b7c69827477a305a454cc8149627cb1e54fdf193d965280e6" Sep 10 22:37:37.723678 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:37.723659 2564 scope.go:117] "RemoveContainer" containerID="509295bdfb24826b7c69827477a305a454cc8149627cb1e54fdf193d965280e6" Sep 10 22:37:37.723945 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:37:37.723922 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"509295bdfb24826b7c69827477a305a454cc8149627cb1e54fdf193d965280e6\": container with ID starting with 509295bdfb24826b7c69827477a305a454cc8149627cb1e54fdf193d965280e6 not found: ID does not exist" containerID="509295bdfb24826b7c69827477a305a454cc8149627cb1e54fdf193d965280e6" Sep 10 22:37:37.724020 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:37.723953 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"509295bdfb24826b7c69827477a305a454cc8149627cb1e54fdf193d965280e6"} err="failed to get container status \"509295bdfb24826b7c69827477a305a454cc8149627cb1e54fdf193d965280e6\": rpc error: code = NotFound desc = could not find container \"509295bdfb24826b7c69827477a305a454cc8149627cb1e54fdf193d965280e6\": container with ID starting with 509295bdfb24826b7c69827477a305a454cc8149627cb1e54fdf193d965280e6 not found: ID does not exist" Sep 10 22:37:37.744074 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:37.744017 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b6f6f558b-jrdwd"] Sep 10 22:37:37.752509 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:37.752487 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5b6f6f558b-jrdwd"] Sep 10 22:37:38.639736 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:38.639703 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b6f6f558b-fhvmj"] Sep 10 22:37:38.640160 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:38.640117 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c9d6a1d-d6fa-4ccd-958b-01997968122b" containerName="manager" Sep 10 22:37:38.640160 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:38.640129 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c9d6a1d-d6fa-4ccd-958b-01997968122b" containerName="manager" Sep 10 22:37:38.640236 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:38.640206 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c9d6a1d-d6fa-4ccd-958b-01997968122b" containerName="manager" Sep 10 22:37:38.644682 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:38.644664 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b6f6f558b-fhvmj" Sep 10 22:37:38.651150 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:38.651125 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 22:37:38.651301 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:38.651155 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zr6dm\"" Sep 10 22:37:38.651301 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:38.651160 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 22:37:38.651301 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:38.651220 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 22:37:38.666754 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:38.666729 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b6f6f558b-fhvmj"] Sep 10 22:37:38.800424 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:38.800384 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k22b4\" (UniqueName: \"kubernetes.io/projected/e851c624-7c32-49dc-93ae-89a4e61b6725-kube-api-access-k22b4\") pod \"maas-controller-5b6f6f558b-fhvmj\" (UID: \"e851c624-7c32-49dc-93ae-89a4e61b6725\") " pod="opendatahub/maas-controller-5b6f6f558b-fhvmj" Sep 10 22:37:38.800424 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:38.800424 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e851c624-7c32-49dc-93ae-89a4e61b6725-webhook-cert\") pod \"maas-controller-5b6f6f558b-fhvmj\" (UID: \"e851c624-7c32-49dc-93ae-89a4e61b6725\") " pod="opendatahub/maas-controller-5b6f6f558b-fhvmj" Sep 10 22:37:38.800646 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:38.800520 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e851c624-7c32-49dc-93ae-89a4e61b6725-metrics-certs\") pod \"maas-controller-5b6f6f558b-fhvmj\" (UID: \"e851c624-7c32-49dc-93ae-89a4e61b6725\") " pod="opendatahub/maas-controller-5b6f6f558b-fhvmj" Sep 10 22:37:38.901118 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:38.901036 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k22b4\" (UniqueName: \"kubernetes.io/projected/e851c624-7c32-49dc-93ae-89a4e61b6725-kube-api-access-k22b4\") pod \"maas-controller-5b6f6f558b-fhvmj\" (UID: \"e851c624-7c32-49dc-93ae-89a4e61b6725\") " pod="opendatahub/maas-controller-5b6f6f558b-fhvmj" Sep 10 22:37:38.901297 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:38.901137 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e851c624-7c32-49dc-93ae-89a4e61b6725-webhook-cert\") pod \"maas-controller-5b6f6f558b-fhvmj\" (UID: \"e851c624-7c32-49dc-93ae-89a4e61b6725\") " pod="opendatahub/maas-controller-5b6f6f558b-fhvmj" Sep 10 22:37:38.901297 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:38.901196 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e851c624-7c32-49dc-93ae-89a4e61b6725-metrics-certs\") pod \"maas-controller-5b6f6f558b-fhvmj\" (UID: \"e851c624-7c32-49dc-93ae-89a4e61b6725\") " pod="opendatahub/maas-controller-5b6f6f558b-fhvmj" Sep 10 22:37:38.903668 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:38.903641 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e851c624-7c32-49dc-93ae-89a4e61b6725-webhook-cert\") pod \"maas-controller-5b6f6f558b-fhvmj\" (UID: \"e851c624-7c32-49dc-93ae-89a4e61b6725\") " pod="opendatahub/maas-controller-5b6f6f558b-fhvmj" Sep 10 22:37:38.903773 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:38.903641 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e851c624-7c32-49dc-93ae-89a4e61b6725-metrics-certs\") pod \"maas-controller-5b6f6f558b-fhvmj\" (UID: \"e851c624-7c32-49dc-93ae-89a4e61b6725\") " pod="opendatahub/maas-controller-5b6f6f558b-fhvmj" Sep 10 22:37:38.914874 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:38.914847 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k22b4\" (UniqueName: \"kubernetes.io/projected/e851c624-7c32-49dc-93ae-89a4e61b6725-kube-api-access-k22b4\") pod \"maas-controller-5b6f6f558b-fhvmj\" (UID: \"e851c624-7c32-49dc-93ae-89a4e61b6725\") " pod="opendatahub/maas-controller-5b6f6f558b-fhvmj" Sep 10 22:37:38.937395 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:38.937357 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c9d6a1d-d6fa-4ccd-958b-01997968122b" path="/var/lib/kubelet/pods/6c9d6a1d-d6fa-4ccd-958b-01997968122b/volumes" Sep 10 22:37:38.954434 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:38.954406 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b6f6f558b-fhvmj" Sep 10 22:37:39.232363 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:37:39.232336 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode851c624_7c32_49dc_93ae_89a4e61b6725.slice/crio-fd1f9343f7c29518f1b660d94bafd1410f3c896eb46ea110d57a825e7c797a6b WatchSource:0}: Error finding container fd1f9343f7c29518f1b660d94bafd1410f3c896eb46ea110d57a825e7c797a6b: Status 404 returned error can't find the container with id fd1f9343f7c29518f1b660d94bafd1410f3c896eb46ea110d57a825e7c797a6b Sep 10 22:37:39.232523 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:39.232502 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b6f6f558b-fhvmj"] Sep 10 22:37:39.727551 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:39.727517 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b6f6f558b-fhvmj" event={"ID":"e851c624-7c32-49dc-93ae-89a4e61b6725","Type":"ContainerStarted","Data":"603068d8e5312c8b7ed12cd7fee5d5f842f7d64656dc0b264877e9baf2c38356"} Sep 10 22:37:39.727889 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:39.727620 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b6f6f558b-fhvmj" event={"ID":"e851c624-7c32-49dc-93ae-89a4e61b6725","Type":"ContainerStarted","Data":"fd1f9343f7c29518f1b660d94bafd1410f3c896eb46ea110d57a825e7c797a6b"} Sep 10 22:37:39.727889 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:39.727657 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b6f6f558b-fhvmj" Sep 10 22:37:39.766597 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:39.766548 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b6f6f558b-fhvmj" podStartSLOduration=1.33821214 podStartE2EDuration="1.766533257s" totalImagesPullingTime="428.321117ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:37:39.765770379 +0000 UTC m=+1961.335260342" watchObservedRunningTime="2026-09-10 22:37:39.766533257 +0000 UTC m=+1961.336023221" Sep 10 22:37:50.735322 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:37:50.735281 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b6f6f558b-fhvmj" Sep 10 22:38:48.764297 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:38:48.764262 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7kmv"] Sep 10 22:38:48.764765 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:38:48.764448 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7kmv" podUID="f8226034-0cc3-4303-9d1b-d42162b6c355" containerName="manager" containerID="cri-o://f94888076ce29e1a75df83f3e3fa7a687d54059bd7f60b0938df87c906d57f34" gracePeriod=10 Sep 10 22:38:51.074155 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:38:51.074130 2564 generic.go:356] "Generic (PLEG): container finished" podID="f8226034-0cc3-4303-9d1b-d42162b6c355" containerID="f94888076ce29e1a75df83f3e3fa7a687d54059bd7f60b0938df87c906d57f34" exitCode=0 Sep 10 22:38:51.074468 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:38:51.074173 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7kmv" event={"ID":"f8226034-0cc3-4303-9d1b-d42162b6c355","Type":"ContainerDied","Data":"f94888076ce29e1a75df83f3e3fa7a687d54059bd7f60b0938df87c906d57f34"} Sep 10 22:38:51.111269 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:38:51.111249 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7kmv" Sep 10 22:38:51.160727 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:38:51.160701 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7csdd\" (UniqueName: \"kubernetes.io/projected/f8226034-0cc3-4303-9d1b-d42162b6c355-kube-api-access-7csdd\") pod \"f8226034-0cc3-4303-9d1b-d42162b6c355\" (UID: \"f8226034-0cc3-4303-9d1b-d42162b6c355\") " Sep 10 22:38:51.160854 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:38:51.160806 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8226034-0cc3-4303-9d1b-d42162b6c355-extensions-socket-volume\") pod \"f8226034-0cc3-4303-9d1b-d42162b6c355\" (UID: \"f8226034-0cc3-4303-9d1b-d42162b6c355\") " Sep 10 22:38:51.161159 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:38:51.161139 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8226034-0cc3-4303-9d1b-d42162b6c355-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f8226034-0cc3-4303-9d1b-d42162b6c355" (UID: "f8226034-0cc3-4303-9d1b-d42162b6c355"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 22:38:51.161248 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:38:51.161237 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f8226034-0cc3-4303-9d1b-d42162b6c355-extensions-socket-volume\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:38:51.162949 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:38:51.162891 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8226034-0cc3-4303-9d1b-d42162b6c355-kube-api-access-7csdd" (OuterVolumeSpecName: "kube-api-access-7csdd") pod "f8226034-0cc3-4303-9d1b-d42162b6c355" (UID: "f8226034-0cc3-4303-9d1b-d42162b6c355"). InnerVolumeSpecName "kube-api-access-7csdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:38:51.262719 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:38:51.262664 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7csdd\" (UniqueName: \"kubernetes.io/projected/f8226034-0cc3-4303-9d1b-d42162b6c355-kube-api-access-7csdd\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:38:52.080033 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:38:52.080001 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7kmv" Sep 10 22:38:52.080463 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:38:52.080038 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7kmv" event={"ID":"f8226034-0cc3-4303-9d1b-d42162b6c355","Type":"ContainerDied","Data":"a91a4d96f02313c84a39a73b3d4e2d6fca28eff6f1a59b034ed456c0fd32bb99"} Sep 10 22:38:52.080463 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:38:52.080111 2564 scope.go:117] "RemoveContainer" containerID="f94888076ce29e1a75df83f3e3fa7a687d54059bd7f60b0938df87c906d57f34" Sep 10 22:38:52.104152 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:38:52.104129 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7kmv"] Sep 10 22:38:52.110312 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:38:52.110290 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-d7kmv"] Sep 10 22:38:52.937067 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:38:52.937035 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8226034-0cc3-4303-9d1b-d42162b6c355" path="/var/lib/kubelet/pods/f8226034-0cc3-4303-9d1b-d42162b6c355/volumes" Sep 10 22:39:49.470503 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:39:49.470467 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xzs2r"] Sep 10 22:39:49.470970 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:39:49.470919 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8226034-0cc3-4303-9d1b-d42162b6c355" containerName="manager" Sep 10 22:39:49.470970 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:39:49.470934 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8226034-0cc3-4303-9d1b-d42162b6c355" containerName="manager" Sep 10 22:39:49.471052 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:39:49.471005 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8226034-0cc3-4303-9d1b-d42162b6c355" containerName="manager" Sep 10 22:39:49.474153 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:39:49.474135 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xzs2r" Sep 10 22:39:49.476913 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:39:49.476876 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-d2hqd\"" Sep 10 22:39:49.494656 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:39:49.494631 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xzs2r"] Sep 10 22:39:49.587149 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:39:49.587109 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a34e47c7-8b6f-41d1-baac-fcbaccc4fc65-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xzs2r\" (UID: \"a34e47c7-8b6f-41d1-baac-fcbaccc4fc65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xzs2r" Sep 10 22:39:49.587338 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:39:49.587255 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsrjl\" (UniqueName: \"kubernetes.io/projected/a34e47c7-8b6f-41d1-baac-fcbaccc4fc65-kube-api-access-zsrjl\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xzs2r\" (UID: \"a34e47c7-8b6f-41d1-baac-fcbaccc4fc65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xzs2r" Sep 10 22:39:49.688351 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:39:49.688312 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsrjl\" (UniqueName: \"kubernetes.io/projected/a34e47c7-8b6f-41d1-baac-fcbaccc4fc65-kube-api-access-zsrjl\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xzs2r\" (UID: \"a34e47c7-8b6f-41d1-baac-fcbaccc4fc65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xzs2r" Sep 10 22:39:49.688558 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:39:49.688381 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a34e47c7-8b6f-41d1-baac-fcbaccc4fc65-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xzs2r\" (UID: \"a34e47c7-8b6f-41d1-baac-fcbaccc4fc65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xzs2r" Sep 10 22:39:49.688747 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:39:49.688724 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a34e47c7-8b6f-41d1-baac-fcbaccc4fc65-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xzs2r\" (UID: \"a34e47c7-8b6f-41d1-baac-fcbaccc4fc65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xzs2r" Sep 10 22:39:49.707116 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:39:49.707079 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsrjl\" (UniqueName: \"kubernetes.io/projected/a34e47c7-8b6f-41d1-baac-fcbaccc4fc65-kube-api-access-zsrjl\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xzs2r\" (UID: \"a34e47c7-8b6f-41d1-baac-fcbaccc4fc65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xzs2r" Sep 10 22:39:49.784285 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:39:49.784190 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xzs2r" Sep 10 22:39:50.064212 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:39:50.064182 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xzs2r"] Sep 10 22:39:50.065467 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:39:50.065438 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda34e47c7_8b6f_41d1_baac_fcbaccc4fc65.slice/crio-a6b5b784a7aada6c46dc37bf595393fc9818a2b20415069571d8f957a8750c08 WatchSource:0}: Error finding container a6b5b784a7aada6c46dc37bf595393fc9818a2b20415069571d8f957a8750c08: Status 404 returned error can't find the container with id a6b5b784a7aada6c46dc37bf595393fc9818a2b20415069571d8f957a8750c08 Sep 10 22:39:50.358054 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:39:50.358017 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xzs2r" event={"ID":"a34e47c7-8b6f-41d1-baac-fcbaccc4fc65","Type":"ContainerStarted","Data":"42c0fab391ec8ef1b657dac4f1c42a9f4c023d9de7a57f6d67bf32748d87a004"} Sep 10 22:39:50.358223 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:39:50.358082 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xzs2r" event={"ID":"a34e47c7-8b6f-41d1-baac-fcbaccc4fc65","Type":"ContainerStarted","Data":"a6b5b784a7aada6c46dc37bf595393fc9818a2b20415069571d8f957a8750c08"} Sep 10 22:39:50.358276 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:39:50.358258 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xzs2r" Sep 10 22:39:50.385203 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:39:50.385146 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xzs2r" podStartSLOduration=1.385130193 podStartE2EDuration="1.385130193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:39:50.383453907 +0000 UTC m=+2091.952943872" watchObservedRunningTime="2026-09-10 22:39:50.385130193 +0000 UTC m=+2091.954620156" Sep 10 22:40:01.364966 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:40:01.364931 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xzs2r" Sep 10 22:41:23.934303 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:23.934274 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-65c67cbdfc-qtbdx"] Sep 10 22:41:23.937575 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:23.937561 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-qtbdx" Sep 10 22:41:23.949483 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:23.949460 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65c67cbdfc-qtbdx"] Sep 10 22:41:23.958019 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:23.957993 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4361940-bdcd-43d2-9450-d6a5098c8b1c-metrics-certs\") pod \"maas-api-65c67cbdfc-qtbdx\" (UID: \"c4361940-bdcd-43d2-9450-d6a5098c8b1c\") " pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-qtbdx" Sep 10 22:41:23.958130 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:23.958113 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kfm6\" (UniqueName: \"kubernetes.io/projected/c4361940-bdcd-43d2-9450-d6a5098c8b1c-kube-api-access-4kfm6\") pod \"maas-api-65c67cbdfc-qtbdx\" (UID: \"c4361940-bdcd-43d2-9450-d6a5098c8b1c\") " pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-qtbdx" Sep 10 22:41:23.958174 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:23.958158 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4361940-bdcd-43d2-9450-d6a5098c8b1c-maas-api-tls\") pod \"maas-api-65c67cbdfc-qtbdx\" (UID: \"c4361940-bdcd-43d2-9450-d6a5098c8b1c\") " pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-qtbdx" Sep 10 22:41:23.958256 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:23.958236 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4361940-bdcd-43d2-9450-d6a5098c8b1c-openshift-service-ca\") pod \"maas-api-65c67cbdfc-qtbdx\" (UID: \"c4361940-bdcd-43d2-9450-d6a5098c8b1c\") " pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-qtbdx" Sep 10 22:41:24.059287 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:24.059253 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4361940-bdcd-43d2-9450-d6a5098c8b1c-openshift-service-ca\") pod \"maas-api-65c67cbdfc-qtbdx\" (UID: \"c4361940-bdcd-43d2-9450-d6a5098c8b1c\") " pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-qtbdx" Sep 10 22:41:24.059479 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:24.059319 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4361940-bdcd-43d2-9450-d6a5098c8b1c-metrics-certs\") pod \"maas-api-65c67cbdfc-qtbdx\" (UID: \"c4361940-bdcd-43d2-9450-d6a5098c8b1c\") " pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-qtbdx" Sep 10 22:41:24.059479 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:24.059363 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kfm6\" (UniqueName: \"kubernetes.io/projected/c4361940-bdcd-43d2-9450-d6a5098c8b1c-kube-api-access-4kfm6\") pod \"maas-api-65c67cbdfc-qtbdx\" (UID: \"c4361940-bdcd-43d2-9450-d6a5098c8b1c\") " pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-qtbdx" Sep 10 22:41:24.059479 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:24.059385 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4361940-bdcd-43d2-9450-d6a5098c8b1c-maas-api-tls\") pod \"maas-api-65c67cbdfc-qtbdx\" (UID: \"c4361940-bdcd-43d2-9450-d6a5098c8b1c\") " pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-qtbdx" Sep 10 22:41:24.060088 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:24.060065 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4361940-bdcd-43d2-9450-d6a5098c8b1c-openshift-service-ca\") pod \"maas-api-65c67cbdfc-qtbdx\" (UID: \"c4361940-bdcd-43d2-9450-d6a5098c8b1c\") " pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-qtbdx" Sep 10 22:41:24.061878 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:24.061858 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4361940-bdcd-43d2-9450-d6a5098c8b1c-metrics-certs\") pod \"maas-api-65c67cbdfc-qtbdx\" (UID: \"c4361940-bdcd-43d2-9450-d6a5098c8b1c\") " pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-qtbdx" Sep 10 22:41:24.062019 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:24.061999 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4361940-bdcd-43d2-9450-d6a5098c8b1c-maas-api-tls\") pod \"maas-api-65c67cbdfc-qtbdx\" (UID: \"c4361940-bdcd-43d2-9450-d6a5098c8b1c\") " pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-qtbdx" Sep 10 22:41:24.067546 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:24.067526 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kfm6\" (UniqueName: \"kubernetes.io/projected/c4361940-bdcd-43d2-9450-d6a5098c8b1c-kube-api-access-4kfm6\") pod \"maas-api-65c67cbdfc-qtbdx\" (UID: \"c4361940-bdcd-43d2-9450-d6a5098c8b1c\") " pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-qtbdx" Sep 10 22:41:24.248362 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:24.248276 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-qtbdx" Sep 10 22:41:24.524303 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:24.524239 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65c67cbdfc-qtbdx"] Sep 10 22:41:24.526709 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:41:24.526680 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4361940_bdcd_43d2_9450_d6a5098c8b1c.slice/crio-852afcf3171dfe49617d98e74096f84c8ed49dac0d6f7d540525d3c17b85ce09 WatchSource:0}: Error finding container 852afcf3171dfe49617d98e74096f84c8ed49dac0d6f7d540525d3c17b85ce09: Status 404 returned error can't find the container with id 852afcf3171dfe49617d98e74096f84c8ed49dac0d6f7d540525d3c17b85ce09 Sep 10 22:41:24.528050 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:24.528032 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 22:41:24.811045 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:24.810965 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-qtbdx" event={"ID":"c4361940-bdcd-43d2-9450-d6a5098c8b1c","Type":"ContainerStarted","Data":"852afcf3171dfe49617d98e74096f84c8ed49dac0d6f7d540525d3c17b85ce09"} Sep 10 22:41:25.817313 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:25.817259 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-qtbdx" event={"ID":"c4361940-bdcd-43d2-9450-d6a5098c8b1c","Type":"ContainerStarted","Data":"173c9fed77a731134dbf27bd6753f35751777318a3d978e53dd16685dc666db8"} Sep 10 22:41:25.817734 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:25.817354 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-qtbdx" Sep 10 22:41:25.835977 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:25.835931 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-qtbdx" podStartSLOduration=2.21275839 podStartE2EDuration="2.835890701s" totalImagesPullingTime="623.132311ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:41:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:41:25.834624944 +0000 UTC m=+2187.404114908" watchObservedRunningTime="2026-09-10 22:41:25.835890701 +0000 UTC m=+2187.405380664" Sep 10 22:41:31.826642 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:31.826614 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-qtbdx" Sep 10 22:41:40.909890 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:40.909843 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-554988677c-lxsxm"] Sep 10 22:41:40.910325 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:40.910149 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-554988677c-lxsxm" podUID="11db8a20-d065-41c8-8fb7-3bc09a778ba3" containerName="maas-api" containerID="cri-o://c72053253f1e0be307197c2eb4b90c19160279bb316188f17298a9d43fdf3a17" gracePeriod=30 Sep 10 22:41:42.158271 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:42.158240 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-554988677c-lxsxm" Sep 10 22:41:42.224798 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:42.224691 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11db8a20-d065-41c8-8fb7-3bc09a778ba3-openshift-service-ca\") pod \"11db8a20-d065-41c8-8fb7-3bc09a778ba3\" (UID: \"11db8a20-d065-41c8-8fb7-3bc09a778ba3\") " Sep 10 22:41:42.225008 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:42.224839 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11db8a20-d065-41c8-8fb7-3bc09a778ba3-metrics-certs\") pod \"11db8a20-d065-41c8-8fb7-3bc09a778ba3\" (UID: \"11db8a20-d065-41c8-8fb7-3bc09a778ba3\") " Sep 10 22:41:42.225008 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:42.224876 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zstx9\" (UniqueName: \"kubernetes.io/projected/11db8a20-d065-41c8-8fb7-3bc09a778ba3-kube-api-access-zstx9\") pod \"11db8a20-d065-41c8-8fb7-3bc09a778ba3\" (UID: \"11db8a20-d065-41c8-8fb7-3bc09a778ba3\") " Sep 10 22:41:42.225008 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:42.224939 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11db8a20-d065-41c8-8fb7-3bc09a778ba3-maas-api-tls\") pod \"11db8a20-d065-41c8-8fb7-3bc09a778ba3\" (UID: \"11db8a20-d065-41c8-8fb7-3bc09a778ba3\") " Sep 10 22:41:42.225008 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:42.225002 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11db8a20-d065-41c8-8fb7-3bc09a778ba3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "11db8a20-d065-41c8-8fb7-3bc09a778ba3" (UID: "11db8a20-d065-41c8-8fb7-3bc09a778ba3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 22:41:42.225285 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:42.225266 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11db8a20-d065-41c8-8fb7-3bc09a778ba3-openshift-service-ca\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:41:42.226963 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:42.226893 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11db8a20-d065-41c8-8fb7-3bc09a778ba3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "11db8a20-d065-41c8-8fb7-3bc09a778ba3" (UID: "11db8a20-d065-41c8-8fb7-3bc09a778ba3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:41:42.226963 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:42.226934 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11db8a20-d065-41c8-8fb7-3bc09a778ba3-kube-api-access-zstx9" (OuterVolumeSpecName: "kube-api-access-zstx9") pod "11db8a20-d065-41c8-8fb7-3bc09a778ba3" (UID: "11db8a20-d065-41c8-8fb7-3bc09a778ba3"). InnerVolumeSpecName "kube-api-access-zstx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 22:41:42.227127 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:42.227038 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11db8a20-d065-41c8-8fb7-3bc09a778ba3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "11db8a20-d065-41c8-8fb7-3bc09a778ba3" (UID: "11db8a20-d065-41c8-8fb7-3bc09a778ba3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 22:41:42.325869 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:42.325775 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11db8a20-d065-41c8-8fb7-3bc09a778ba3-maas-api-tls\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:41:42.325869 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:42.325808 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11db8a20-d065-41c8-8fb7-3bc09a778ba3-metrics-certs\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:41:42.325869 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:42.325823 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zstx9\" (UniqueName: \"kubernetes.io/projected/11db8a20-d065-41c8-8fb7-3bc09a778ba3-kube-api-access-zstx9\") on node \"ip-10-0-137-148.ec2.internal\" DevicePath \"\"" Sep 10 22:41:42.905694 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:42.905658 2564 generic.go:356] "Generic (PLEG): container finished" podID="11db8a20-d065-41c8-8fb7-3bc09a778ba3" containerID="c72053253f1e0be307197c2eb4b90c19160279bb316188f17298a9d43fdf3a17" exitCode=0 Sep 10 22:41:42.905922 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:42.905706 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-554988677c-lxsxm" Sep 10 22:41:42.905922 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:42.905744 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-554988677c-lxsxm" event={"ID":"11db8a20-d065-41c8-8fb7-3bc09a778ba3","Type":"ContainerDied","Data":"c72053253f1e0be307197c2eb4b90c19160279bb316188f17298a9d43fdf3a17"} Sep 10 22:41:42.905922 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:42.905832 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-554988677c-lxsxm" event={"ID":"11db8a20-d065-41c8-8fb7-3bc09a778ba3","Type":"ContainerDied","Data":"6abacaf3765f6fe1829f3b6ddea52d805272c4aa48a384bc660567deb48794e4"} Sep 10 22:41:42.905922 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:42.905862 2564 scope.go:117] "RemoveContainer" containerID="c72053253f1e0be307197c2eb4b90c19160279bb316188f17298a9d43fdf3a17" Sep 10 22:41:42.915152 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:42.915131 2564 scope.go:117] "RemoveContainer" containerID="c72053253f1e0be307197c2eb4b90c19160279bb316188f17298a9d43fdf3a17" Sep 10 22:41:42.915445 ip-10-0-137-148 kubenswrapper[2564]: E0910 22:41:42.915427 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c72053253f1e0be307197c2eb4b90c19160279bb316188f17298a9d43fdf3a17\": container with ID starting with c72053253f1e0be307197c2eb4b90c19160279bb316188f17298a9d43fdf3a17 not found: ID does not exist" containerID="c72053253f1e0be307197c2eb4b90c19160279bb316188f17298a9d43fdf3a17" Sep 10 22:41:42.915491 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:42.915454 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c72053253f1e0be307197c2eb4b90c19160279bb316188f17298a9d43fdf3a17"} err="failed to get container status \"c72053253f1e0be307197c2eb4b90c19160279bb316188f17298a9d43fdf3a17\": rpc error: code = NotFound desc = could not find container \"c72053253f1e0be307197c2eb4b90c19160279bb316188f17298a9d43fdf3a17\": container with ID starting with c72053253f1e0be307197c2eb4b90c19160279bb316188f17298a9d43fdf3a17 not found: ID does not exist" Sep 10 22:41:42.938269 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:42.938245 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-554988677c-lxsxm"] Sep 10 22:41:42.938269 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:42.938275 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-554988677c-lxsxm"] Sep 10 22:41:44.937642 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:44.937606 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11db8a20-d065-41c8-8fb7-3bc09a778ba3" path="/var/lib/kubelet/pods/11db8a20-d065-41c8-8fb7-3bc09a778ba3/volumes" Sep 10 22:41:53.927612 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:53.927575 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5474b4d86d-hqp85_4f72c97e-e212-4128-9251-35fe4ca5fdb3/authorino/0.log" Sep 10 22:41:59.628461 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:59.628405 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tv2jc_6dd29f4d-c5ca-43b3-97d4-130855c684b0/manager/2.log" Sep 10 22:41:59.756216 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:41:59.756181 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b6f6f558b-fhvmj_e851c624-7c32-49dc-93ae-89a4e61b6725/manager/0.log" Sep 10 22:42:00.379105 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:00.379074 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-995c55587-hppcq_cee352bc-150c-4c82-901a-e5871cbf837d/copy-manifests/0.log" Sep 10 22:42:00.396962 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:00.396930 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-995c55587-hppcq_cee352bc-150c-4c82-901a-e5871cbf837d/manager/0.log" Sep 10 22:42:00.760650 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:00.760565 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-65c67cbdfc-qtbdx_c4361940-bdcd-43d2-9450-d6a5098c8b1c/maas-api/0.log" Sep 10 22:42:01.879200 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:01.879159 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5474b4d86d-hqp85_4f72c97e-e212-4128-9251-35fe4ca5fdb3/authorino/0.log" Sep 10 22:42:02.334414 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:02.334338 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-xzs2r_a34e47c7-8b6f-41d1-baac-fcbaccc4fc65/manager/0.log" Sep 10 22:42:03.110648 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:03.110615 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9b867fd64-b7wgk_5062cdac-ad2e-4b2c-b65c-0a64478e4ec6/kube-auth-proxy/0.log" Sep 10 22:42:04.148512 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:04.148481 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-kt72n_73cadc3a-00f0-41b0-9e8d-9a0839c6800e/main/0.log" Sep 10 22:42:04.494117 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:04.494018 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fflbwj_fc904b9f-ab74-48db-9212-324873622c3d/main/0.log" Sep 10 22:42:13.397488 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:13.397452 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m6nqn_673499d4-211d-40e2-8a3f-95c74bc1d0f3/global-pull-secret-syncer/0.log" Sep 10 22:42:13.538876 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:13.538830 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-96t46_4f2170c9-cacb-47f3-8a42-b80039deefef/konnectivity-agent/0.log" Sep 10 22:42:13.604322 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:13.604290 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-148.ec2.internal_f09bb482ad5b644a06b2ecf966b25fc2/haproxy/0.log" Sep 10 22:42:18.702227 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:18.702194 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5474b4d86d-hqp85_4f72c97e-e212-4128-9251-35fe4ca5fdb3/authorino/0.log" Sep 10 22:42:18.890781 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:18.890723 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-xzs2r_a34e47c7-8b6f-41d1-baac-fcbaccc4fc65/manager/0.log" Sep 10 22:42:20.461156 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:20.461117 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f978096-abb4-4903-8824-7ed487f9d0b3/alertmanager/0.log" Sep 10 22:42:20.488177 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:20.488143 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f978096-abb4-4903-8824-7ed487f9d0b3/config-reloader/0.log" Sep 10 22:42:20.525123 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:20.525094 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f978096-abb4-4903-8824-7ed487f9d0b3/kube-rbac-proxy-web/0.log" Sep 10 22:42:20.550197 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:20.550163 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f978096-abb4-4903-8824-7ed487f9d0b3/kube-rbac-proxy/0.log" Sep 10 22:42:20.579274 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:20.579248 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f978096-abb4-4903-8824-7ed487f9d0b3/kube-rbac-proxy-metric/0.log" Sep 10 22:42:20.599443 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:20.599363 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f978096-abb4-4903-8824-7ed487f9d0b3/prom-label-proxy/0.log" Sep 10 22:42:20.622353 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:20.622330 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f978096-abb4-4903-8824-7ed487f9d0b3/init-config-reloader/0.log" Sep 10 22:42:20.675605 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:20.675577 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-fpgjg_1086cb8d-b834-4e42-87df-6db5b0bd9037/cluster-monitoring-operator/0.log" Sep 10 22:42:20.778389 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:20.778359 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-748b5b89c4-pljql_cdf04798-bfc9-437a-8ed0-e3d893748ed0/metrics-server/0.log" Sep 10 22:42:20.820309 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:20.820280 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-k77xr_1d55f82c-3a6d-4393-824e-8197bb503f20/monitoring-plugin/0.log" Sep 10 22:42:21.040037 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:21.040006 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g575r_2d70dda5-31b5-4c42-852f-83030940374f/node-exporter/0.log" Sep 10 22:42:21.067074 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:21.067044 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g575r_2d70dda5-31b5-4c42-852f-83030940374f/kube-rbac-proxy/0.log" Sep 10 22:42:21.087454 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:21.087429 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g575r_2d70dda5-31b5-4c42-852f-83030940374f/init-textfile/0.log" Sep 10 22:42:21.237299 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:21.237175 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c8e571e-6e80-4e58-ac7b-8e09b9d52218/prometheus/0.log" Sep 10 22:42:21.260521 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:21.260495 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c8e571e-6e80-4e58-ac7b-8e09b9d52218/config-reloader/0.log" Sep 10 22:42:21.282282 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:21.282254 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c8e571e-6e80-4e58-ac7b-8e09b9d52218/thanos-sidecar/0.log" Sep 10 22:42:21.306175 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:21.306146 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c8e571e-6e80-4e58-ac7b-8e09b9d52218/kube-rbac-proxy-web/0.log" Sep 10 22:42:21.328086 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:21.328061 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c8e571e-6e80-4e58-ac7b-8e09b9d52218/kube-rbac-proxy/0.log" Sep 10 22:42:21.352965 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:21.352942 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c8e571e-6e80-4e58-ac7b-8e09b9d52218/kube-rbac-proxy-thanos/0.log" Sep 10 22:42:21.379815 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:21.379787 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c8e571e-6e80-4e58-ac7b-8e09b9d52218/init-config-reloader/0.log" Sep 10 22:42:21.463173 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:21.463136 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-lvr27_40779462-2044-46c5-9c26-9dd79bd26267/prometheus-operator-admission-webhook/0.log" Sep 10 22:42:21.494260 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:21.494185 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67fd576bb5-btwmm_4b8d78cd-db2b-48ae-a85e-78621053717b/telemeter-client/0.log" Sep 10 22:42:21.516358 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:21.516331 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67fd576bb5-btwmm_4b8d78cd-db2b-48ae-a85e-78621053717b/reload/0.log" Sep 10 22:42:21.541568 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:21.541537 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67fd576bb5-btwmm_4b8d78cd-db2b-48ae-a85e-78621053717b/kube-rbac-proxy/0.log" Sep 10 22:42:21.573910 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:21.573860 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fdd7844f8-89nmk_24cec8b0-92f4-4f81-be7b-7ca9df75bdf8/thanos-query/0.log" Sep 10 22:42:21.594616 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:21.594591 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fdd7844f8-89nmk_24cec8b0-92f4-4f81-be7b-7ca9df75bdf8/kube-rbac-proxy-web/0.log" Sep 10 22:42:21.615527 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:21.615506 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fdd7844f8-89nmk_24cec8b0-92f4-4f81-be7b-7ca9df75bdf8/kube-rbac-proxy/0.log" Sep 10 22:42:21.637336 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:21.637315 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fdd7844f8-89nmk_24cec8b0-92f4-4f81-be7b-7ca9df75bdf8/prom-label-proxy/0.log" Sep 10 22:42:21.668255 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:21.668220 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fdd7844f8-89nmk_24cec8b0-92f4-4f81-be7b-7ca9df75bdf8/kube-rbac-proxy-rules/0.log" Sep 10 22:42:21.687912 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:21.687885 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fdd7844f8-89nmk_24cec8b0-92f4-4f81-be7b-7ca9df75bdf8/kube-rbac-proxy-metrics/0.log" Sep 10 22:42:22.772438 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:22.772405 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-5fl5b_b21f2cfa-98bc-4a1b-9f4a-747ef2a45c50/networking-console-plugin/0.log" Sep 10 22:42:23.057219 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.057144 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f4kkc/perf-node-gather-daemonset-vkvlx"] Sep 10 22:42:23.057547 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.057536 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11db8a20-d065-41c8-8fb7-3bc09a778ba3" containerName="maas-api" Sep 10 22:42:23.057593 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.057549 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="11db8a20-d065-41c8-8fb7-3bc09a778ba3" containerName="maas-api" Sep 10 22:42:23.057627 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.057606 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="11db8a20-d065-41c8-8fb7-3bc09a778ba3" containerName="maas-api" Sep 10 22:42:23.060779 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.060751 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f4kkc/perf-node-gather-daemonset-vkvlx" Sep 10 22:42:23.063630 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.063613 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f4kkc\"/\"openshift-service-ca.crt\"" Sep 10 22:42:23.064802 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.064781 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-f4kkc\"/\"default-dockercfg-8m4xn\"" Sep 10 22:42:23.064953 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.064789 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f4kkc\"/\"kube-root-ca.crt\"" Sep 10 22:42:23.069466 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.069394 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f4kkc/perf-node-gather-daemonset-vkvlx"] Sep 10 22:42:23.119981 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.119944 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lmw9\" (UniqueName: \"kubernetes.io/projected/4a0374fc-31ed-471c-8776-7b321d060b9c-kube-api-access-5lmw9\") pod \"perf-node-gather-daemonset-vkvlx\" (UID: \"4a0374fc-31ed-471c-8776-7b321d060b9c\") " pod="openshift-must-gather-f4kkc/perf-node-gather-daemonset-vkvlx" Sep 10 22:42:23.119981 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.119985 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4a0374fc-31ed-471c-8776-7b321d060b9c-lib-modules\") pod \"perf-node-gather-daemonset-vkvlx\" (UID: \"4a0374fc-31ed-471c-8776-7b321d060b9c\") " pod="openshift-must-gather-f4kkc/perf-node-gather-daemonset-vkvlx" Sep 10 22:42:23.120199 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.120018 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4a0374fc-31ed-471c-8776-7b321d060b9c-podres\") pod \"perf-node-gather-daemonset-vkvlx\" (UID: \"4a0374fc-31ed-471c-8776-7b321d060b9c\") " pod="openshift-must-gather-f4kkc/perf-node-gather-daemonset-vkvlx" Sep 10 22:42:23.120199 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.120051 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a0374fc-31ed-471c-8776-7b321d060b9c-sys\") pod \"perf-node-gather-daemonset-vkvlx\" (UID: \"4a0374fc-31ed-471c-8776-7b321d060b9c\") " pod="openshift-must-gather-f4kkc/perf-node-gather-daemonset-vkvlx" Sep 10 22:42:23.120199 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.120071 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4a0374fc-31ed-471c-8776-7b321d060b9c-proc\") pod \"perf-node-gather-daemonset-vkvlx\" (UID: \"4a0374fc-31ed-471c-8776-7b321d060b9c\") " pod="openshift-must-gather-f4kkc/perf-node-gather-daemonset-vkvlx" Sep 10 22:42:23.221015 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.220979 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lmw9\" (UniqueName: \"kubernetes.io/projected/4a0374fc-31ed-471c-8776-7b321d060b9c-kube-api-access-5lmw9\") pod \"perf-node-gather-daemonset-vkvlx\" (UID: \"4a0374fc-31ed-471c-8776-7b321d060b9c\") " pod="openshift-must-gather-f4kkc/perf-node-gather-daemonset-vkvlx" Sep 10 22:42:23.221198 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.221021 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4a0374fc-31ed-471c-8776-7b321d060b9c-lib-modules\") pod \"perf-node-gather-daemonset-vkvlx\" (UID: \"4a0374fc-31ed-471c-8776-7b321d060b9c\") " pod="openshift-must-gather-f4kkc/perf-node-gather-daemonset-vkvlx" Sep 10 22:42:23.221198 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.221049 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4a0374fc-31ed-471c-8776-7b321d060b9c-podres\") pod \"perf-node-gather-daemonset-vkvlx\" (UID: \"4a0374fc-31ed-471c-8776-7b321d060b9c\") " pod="openshift-must-gather-f4kkc/perf-node-gather-daemonset-vkvlx" Sep 10 22:42:23.221198 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.221085 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a0374fc-31ed-471c-8776-7b321d060b9c-sys\") pod \"perf-node-gather-daemonset-vkvlx\" (UID: \"4a0374fc-31ed-471c-8776-7b321d060b9c\") " pod="openshift-must-gather-f4kkc/perf-node-gather-daemonset-vkvlx" Sep 10 22:42:23.221198 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.221099 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4a0374fc-31ed-471c-8776-7b321d060b9c-proc\") pod \"perf-node-gather-daemonset-vkvlx\" (UID: \"4a0374fc-31ed-471c-8776-7b321d060b9c\") " pod="openshift-must-gather-f4kkc/perf-node-gather-daemonset-vkvlx" Sep 10 22:42:23.221198 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.221187 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4a0374fc-31ed-471c-8776-7b321d060b9c-proc\") pod \"perf-node-gather-daemonset-vkvlx\" (UID: \"4a0374fc-31ed-471c-8776-7b321d060b9c\") " pod="openshift-must-gather-f4kkc/perf-node-gather-daemonset-vkvlx" Sep 10 22:42:23.221198 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.221191 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4a0374fc-31ed-471c-8776-7b321d060b9c-lib-modules\") pod \"perf-node-gather-daemonset-vkvlx\" (UID: \"4a0374fc-31ed-471c-8776-7b321d060b9c\") " pod="openshift-must-gather-f4kkc/perf-node-gather-daemonset-vkvlx" Sep 10 22:42:23.221425 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.221224 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a0374fc-31ed-471c-8776-7b321d060b9c-sys\") pod \"perf-node-gather-daemonset-vkvlx\" (UID: \"4a0374fc-31ed-471c-8776-7b321d060b9c\") " pod="openshift-must-gather-f4kkc/perf-node-gather-daemonset-vkvlx" Sep 10 22:42:23.221425 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.221228 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4a0374fc-31ed-471c-8776-7b321d060b9c-podres\") pod \"perf-node-gather-daemonset-vkvlx\" (UID: \"4a0374fc-31ed-471c-8776-7b321d060b9c\") " pod="openshift-must-gather-f4kkc/perf-node-gather-daemonset-vkvlx" Sep 10 22:42:23.229871 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.229845 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lmw9\" (UniqueName: \"kubernetes.io/projected/4a0374fc-31ed-471c-8776-7b321d060b9c-kube-api-access-5lmw9\") pod \"perf-node-gather-daemonset-vkvlx\" (UID: \"4a0374fc-31ed-471c-8776-7b321d060b9c\") " pod="openshift-must-gather-f4kkc/perf-node-gather-daemonset-vkvlx" Sep 10 22:42:23.371076 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.371048 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f4kkc/perf-node-gather-daemonset-vkvlx" Sep 10 22:42:23.643088 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.643010 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f4kkc/perf-node-gather-daemonset-vkvlx"] Sep 10 22:42:23.646378 ip-10-0-137-148 kubenswrapper[2564]: W0910 22:42:23.646338 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4a0374fc_31ed_471c_8776_7b321d060b9c.slice/crio-7417b786b9f10b79a1b12398566a2ca2c78619f5a8a8e5857148eb28c6e01e60 WatchSource:0}: Error finding container 7417b786b9f10b79a1b12398566a2ca2c78619f5a8a8e5857148eb28c6e01e60: Status 404 returned error can't find the container with id 7417b786b9f10b79a1b12398566a2ca2c78619f5a8a8e5857148eb28c6e01e60 Sep 10 22:42:23.973988 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:23.973893 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-zhmbg_4350b3b2-bbff-447f-b23a-124cc24aebef/download-server/0.log" Sep 10 22:42:24.112106 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:24.112074 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f4kkc/perf-node-gather-daemonset-vkvlx" event={"ID":"4a0374fc-31ed-471c-8776-7b321d060b9c","Type":"ContainerStarted","Data":"95ae6174f0774fe933e2bf34acfe76d1d65522f38ff56bbe7bc1a04056f916de"} Sep 10 22:42:24.112288 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:24.112133 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f4kkc/perf-node-gather-daemonset-vkvlx" event={"ID":"4a0374fc-31ed-471c-8776-7b321d060b9c","Type":"ContainerStarted","Data":"7417b786b9f10b79a1b12398566a2ca2c78619f5a8a8e5857148eb28c6e01e60"} Sep 10 22:42:24.112288 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:24.112215 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-f4kkc/perf-node-gather-daemonset-vkvlx" Sep 10 22:42:24.130428 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:24.130379 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f4kkc/perf-node-gather-daemonset-vkvlx" podStartSLOduration=1.130364165 podStartE2EDuration="1.130364165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:42:24.129076263 +0000 UTC m=+2245.698566228" watchObservedRunningTime="2026-09-10 22:42:24.130364165 +0000 UTC m=+2245.699854128" Sep 10 22:42:25.800206 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:25.800172 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lqltl_0676c0bc-49e3-416a-97eb-70199abcf210/dns/0.log" Sep 10 22:42:25.825594 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:25.825573 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lqltl_0676c0bc-49e3-416a-97eb-70199abcf210/kube-rbac-proxy/0.log" Sep 10 22:42:25.942662 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:25.942631 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zgwrg_0638522d-7b41-42c0-a93c-61fe7e00935b/dns-node-resolver/0.log" Sep 10 22:42:26.480305 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:26.480277 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ph2dv_c7114320-6f0b-4cc6-85a8-309ddda645af/node-ca/0.log" Sep 10 22:42:27.700659 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:27.700629 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9b867fd64-b7wgk_5062cdac-ad2e-4b2c-b65c-0a64478e4ec6/kube-auth-proxy/0.log" Sep 10 22:42:28.752982 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:28.752955 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mp766_643143f3-0560-433b-9fca-05ef72574ee2/serve-healthcheck-canary/0.log" Sep 10 22:42:29.250599 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:29.250564 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-hq6mn_c2b50e35-7ca5-420b-a788-73f839011390/insights-operator/0.log" Sep 10 22:42:29.250942 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:29.250921 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-hq6mn_c2b50e35-7ca5-420b-a788-73f839011390/insights-operator/1.log" Sep 10 22:42:29.270816 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:29.270790 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7bs46_57656f7b-5fcb-46ed-9b3d-63296c4650d8/kube-rbac-proxy/0.log" Sep 10 22:42:29.290370 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:29.290341 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7bs46_57656f7b-5fcb-46ed-9b3d-63296c4650d8/exporter/0.log" Sep 10 22:42:29.310694 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:29.310666 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7bs46_57656f7b-5fcb-46ed-9b3d-63296c4650d8/extractor/0.log" Sep 10 22:42:30.129488 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:30.129453 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-f4kkc/perf-node-gather-daemonset-vkvlx" Sep 10 22:42:31.854829 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:31.854787 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tv2jc_6dd29f4d-c5ca-43b3-97d4-130855c684b0/manager/1.log" Sep 10 22:42:31.901649 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:31.901615 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tv2jc_6dd29f4d-c5ca-43b3-97d4-130855c684b0/manager/2.log" Sep 10 22:42:31.967441 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:31.967411 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b6f6f558b-fhvmj_e851c624-7c32-49dc-93ae-89a4e61b6725/manager/0.log" Sep 10 22:42:32.305483 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:32.305399 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-995c55587-hppcq_cee352bc-150c-4c82-901a-e5871cbf837d/manager/0.log" Sep 10 22:42:32.322243 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:32.322221 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-995c55587-hppcq_cee352bc-150c-4c82-901a-e5871cbf837d/copy-manifests/0.log" Sep 10 22:42:33.656165 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:33.656139 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-578b4977d8-d9snc_7fb31c55-8407-49ea-9b6f-044e2cf97e12/manager/0.log" Sep 10 22:42:34.028330 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:34.028299 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9b867fd64-b7wgk_5062cdac-ad2e-4b2c-b65c-0a64478e4ec6/kube-auth-proxy/0.log" Sep 10 22:42:35.872638 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:35.872605 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9b867fd64-b7wgk_5062cdac-ad2e-4b2c-b65c-0a64478e4ec6/kube-auth-proxy/0.log" Sep 10 22:42:40.152437 ip-10-0-137-148 kubenswrapper[2564]: I0910 22:42:40.152402 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-95n7s_039f078c-f211-4f4f-8878-d39a9e529672/kube-multus/0.log"